Glossary
RAG
Also: Retrieval Augmented Generation
A language model receives matching excerpts from your own data before answering, instead of relying on its training.
The process has two halves. In preparation, documents are split, translated into vectors and stored. At runtime the question is translated as well, the most similar excerpts are retrieved and sent to the model together with the question. The model answers on the basis of what is in the context.
The appeal is obvious: the model knows content it has never seen, updates need no training, and you can cite sources. The price is equally obvious but often underestimated: answer quality depends almost entirely on whether the right excerpts were found.
That shifts the problem from AI to search. Poor results in a RAG system almost never originate in the model but in the splitting, the retrieval or the ranking. Swapping the model when answers are bad treats the symptom.
Pure vector search finds semantically similar text and fails on exact terms: article numbers, legal paragraphs, error codes, proper names. In practice a combination of vector search and classic keyword search is therefore almost always right, followed by reranking.
The second cost factor is context. Every excerpt sent is paid for and distracts at the same time: more context is not better, and past a point it is worse, because the model has to find the relevant passage among the padding.
And finally, permissions. If an employee sees documents through the search that they would not be allowed to see in the source system, that is not an AI problem but a data breach. The permission check belongs in the search, not in the prompt.
How you notice it
- Answers sound plausible but hit the wrong document version.
- Searches for article numbers or legal paragraphs return nothing useful.
- The documents were cut at a fixed character count.
- There is no fixed question set used to check changes.
Not to be confused with
- Fine-tuning
- Changes the behaviour of the model, not its knowledge of current content. For "know our documents" RAG is right, for "answer in our style" fine-tuning is closer.
- Long context window
- Pushing everything in works for small collections and becomes expensive and imprecise once there are many documents. RAG selects instead of sending everything.
- Full-text search
- Finds exact terms reliably but no paraphrases. RAG without a keyword component fails exactly there, which is why the combination is the rule.
- Vector database
- A building block, not the method. Many existing databases handle vectors; a dedicated product only pays off at a certain scale.
When it fits
- Answers should rest on your own, changing content.
- Source citations are required, for example for verifiability.
- The collection is too large to fit into the context in full.
When it does not
- With few, small and stable documents: then the context alone is enough.
- When the gap is not knowledge but the ability to act: that is what tools and MCP are for.
- As long as the source documents are unstructured, outdated or contradictory. RAG amplifies bad content, it does not repair it.
How to approach it
- Clean the sources before indexingRemove outdated versions, duplicates and drafts. A RAG system that finds three contradictory versions of a policy answers wrongly and sounds convincing doing it.
- Split sensiblyCut along structure, not by character count: section, chapter, paragraph. With overlap, so that context does not end mid-sentence.
- Search hybridVector search for meaning, keyword search for numbers and proper names, then merge. The single largest quality jump in most projects.
- Use rerankingLet a reranker pick the best five out of twenty candidates. It costs one extra step and clearly improves the hit rate, because similarity is not relevance.
- Put permissions into the searchFilter before the model, not through an instruction in the prompt. A prompt is not access control.
- Measure with a fixed test setQuestions with a known correct source, run regularly. Without that set nobody notices when a change to the splitting has lowered quality.
Frequently asked
Why does our RAG system answer wrongly although the document exists?
In most cases the right excerpt was not found or not ranked high enough. Check in this order: is the excerpt in the index at all, does the search find it, does it land among the top hits, and is it complete enough to answer the question.
Do you need a vector database for this?
Not necessarily. Many databases already in use can handle vectors these days. A dedicated product pays off with large collections, high query load or special filtering requirements. An additional system also means additional operations.
RAG or fine-tuning?
RAG for knowledge, fine-tuning for behaviour. If the model should know content that changes, RAG is right, because an update only touches the index. If it should answer in a particular format or tone, fine-tuning can be the better choice. Both together is possible and rarely necessary.
How do you handle permissions?
Filtering belongs in the search, before the model. Either index per permission group or filter at query time based on the user identity. An instruction in the prompt not to use certain content is not access control: the document was already in the context by then.