Workbench 01 · Retrieval, measured
Module 12 explains retrieval. Here you run it: split a document into chunks, search it three ways, see exactly which passages an answer would be built from, and test the whole setup against questions with known answers, including questions the document cannot answer. Everything runs on your device.
Each sample document comes with ten questions: eight whose answer is in the text and two whose answer is not. A configuration scores a hit when a retrieved passage contains the answer, and a correct refusal when, for an unanswerable question, nothing clears the confidence threshold.
Why chunk at all
A language model can only take in so much text at once, and the more you give it, the more it gets distracted. Retrieval hands it the few passages that matter. Slide the chunk size and watch an answer get cut in half at small sizes, or buried in a long passage at large ones.
Keyword against meaning
BM25 counts shared words, so it nails names and numbers but misses “profit” when the text says “net income”. Embeddings match meaning, but a passage can sound related without holding the answer. Hybrid ranks by both, which is what most production systems do.
Saying “I don’t know”
The two unanswerable questions in each test set are the most important ones. Set the confidence too low and the system answers everything, inventing support from the nearest passage. Too high and it refuses questions it could answer. The test table shows the trade-off.