RAG: from documents to checkable answers

Give the model evidence at answer time

Retrieval-augmented generation, or RAG, combines a retrieval step with generation. A system finds relevant material and includes it in the model’s input. That is different from retraining the model on your documents.

For a first exercise, use a handful of short public documents. Small inputs let you read the evidence yourself. You do not need a large database to learn why a correct-looking answer can still come from the wrong passage.

Keep retrieval and answering separate

A typical workflow splits documents into useful pieces, indexes them, retrieves candidates for a question and passes selected context to the model. Embeddings are one way to retrieve similar material; keyword search can also be useful. Reranking can reorder retrieved candidates before generation.

Each stage can fail independently. A missing answer may mean the relevant passage was never retrieved. Alternatively, the passage may be present and the generated answer may misread it. Inspect the context before blaming the model.

Try a tiny campus-policy dataset

Write three invented documents: library opening hours, equipment borrowing rules and workshop registration. Give every paragraph a stable source identifier. Make one policy differ between weekdays and weekends. Then write questions that require each distinction.

Start with manual retrieval: choose the paragraph yourself and pass it with the question. If the model still struggles, fixing search will not fix the entire problem. Once the answer stage works on those cases, automate retrieval and compare the results.

Question: Can equipment be borrowed over the weekend?
Retrieved sources:
[borrow-02] Weekend borrowing requires a Friday reservation.

Answer requirement:
State what the source supports and cite its source identifier.
If the evidence is missing, say the documents do not answer it.

Test the evidence, not just the citation

A source label can be copied without supporting the sentence beside it. Check whether each factual claim follows from the cited text. Then ask a question your dataset cannot answer and inspect whether the system invents a policy.

Try two versions of a document with different dates. Decide which is current and make that visible in the retrieved context. Versioning is part of the application’s responsibility; a model cannot reliably resolve a policy conflict that you have left unexplained.

Keep a two-column failure log

Separate retrieval failures from answer failures. In the first column, record whether the necessary passage appeared in the supplied context. In the second, record whether the answer used it correctly. This small distinction tells you where to spend your next hour.

Finish by showing one successful example, one missing-evidence example and one failure you have not fixed. That is a more informative project demonstration than a long chat transcript containing only successful questions.