What Is Retrieval-Augmented Generation? How RAG Works

Key takeaways

• RAG retrieves external information and supplies it as context for a model to generate an answer.
• Retrieval can use keywords, vectors, filters, database queries, or combinations of methods.
• Current documents, preserved qualifications, and enforced access rules matter as much as the generation step.
• Evaluate whether the system found the right evidence and whether the answer accurately used it.

Retrieval-augmented generation (RAG) is an approach that combines retrieving information with generating an answer. The application finds relevant material, supplies it to a language model, and asks the model to respond using that context.

This is useful when a question depends on private documents, recent changes, or detailed information that should not be left to the model’s training alone.

Suppose an employee asks whether a contractor can access a staging database. The answer depends on the organization’s current access policy. RAG can retrieve that policy before generating a response.

The 2020 RAG paper explored combining retrieval with generation for knowledge-intensive language tasks. The broader application pattern now encompasses different retrieval systems and document sources.

How RAG Works

A typical document-based system has a preparation path and a question-answering path.

Preparation:
Documents -> extract content -> divide into passages -> index

Answering:
Question -> retrieve relevant evidence -> assemble context -> generate answer

Preparation makes the content searchable. At query time, the application selects evidence relevant to the question rather than sending every document to the model.

Microsoft’s RAG overview describes how retrieval supports generation over organizational content.

For the contractor question, useful evidence might include eligibility rules, the approval process, and any staging-specific restrictions. A passage about employee access to production would not answer the same question.

Prepare Sources That Can Be Traced and Updated

The index needs more than extracted text. Keep source identifiers, section titles, document versions, and relevant access information associated with the content.

These fields help answer practical questions: which policy did the system use, was it still current, and was the employee allowed to see it?

Index freshness requires an update process. Editing the original policy does not automatically update every extracted passage or cached result. Deleted or superseded documents also need to stop appearing as current evidence.

For our example, an old policy allowing general contractor access could produce a convincing but obsolete answer after a new approval requirement is introduced.

Chunking Must Preserve Qualifications

Long documents are often divided into smaller passages, or chunks, for retrieval.

Suppose a policy says:

Contractors may access staging databases with platform-team approval. Access is limited to synthetic data and expires after seven days.

Splitting the first sentence from the second risks losing the conditions that make the answer correct.

A good chunking strategy preserves meaningful units such as sections, paragraphs, or table rows together with the headings needed to interpret them. The right boundaries depend on the documents and the questions.

This does not mean every chunk should be large. Returning a whole handbook for a narrow question can bury the useful passage among unrelated material.

Does RAG Need a Vector Database?

No. Vector search is one retrieval technique, not a requirement of RAG.

Embeddings represent content numerically so that similarity can help locate related wording. Keyword search can be valuable for exact identifiers, product names, and error codes. A hybrid approach combines those signals; metadata filters can narrow candidates to the right product or document version.

A system could also retrieve structured records through a database query. The defining step is that retrieved information becomes context for generation.

For the access-policy example, the phrase “temporary database credentials” might be relevant even when the employee asks about “contractor access.” Conversely, an exact database name may be too important to rely only on broad similarity.

Generate an Answer That Preserves the Evidence

After retrieval, the application supplies the selected material and the question to the model. Instructions can ask it to identify missing information and cite the passages supporting its claims.

A useful answer in our example would retain all the conditions: approval, synthetic data, and expiration. Simply saying “yes, contractors can access staging” loses essential meaning.

Citations help people inspect that relationship, but their presence is not proof. A real document can be attached to a claim that it does not support.

If the retrieved documents disagree or do not address the question, the system should surface that gap rather than manufacture a definite policy.

Enforce Access Before Supplying Context

The application should retrieve or retain only content the requesting user is permitted to access before assembling model input.

Putting a restricted document into context and asking the model not to reveal it is not an authorization mechanism. The same access boundaries need to apply to source links, cached answers, and any tools used during the interaction.

Retrieved documents are also data, not trusted application instructions. Text inside a document should not gain authority to change tool permissions or request unrelated actions.

These are application-design requirements. Adding retrieval does not automatically implement them.

RAG vs Fine-Tuning

RAG changes the information available when the model answers. Fine-tuning changes model parameters through additional training.

Requirement	Approach to consider
Answer using frequently updated policies	Retrieval with a reliable update process
Show the source behind a claim	Retrieval with explicit source tracking
Improve a repeated task or output behavior	Prompt design, evaluation, and potentially fine-tuning
Work with one short supplied document	Directly provide the document as context

The approaches can be combined. A large context window also does not remove the need to select current, relevant, authorized evidence.

For background on training, see what machine learning is.

Evaluate Retrieval and Generation Separately

Create representative questions with expected evidence and answer criteria. Include a question whose answer is absent, one involving an obsolete policy, and one where the user lacks access.

First check whether retrieval supplied the needed passages. Then check whether generation preserved their meaning. This distinguishes a search failure from a model misreading good evidence.

For the contractor question, an answer omitting the seven-day limit could arise because the limit was not retrieved or because the model ignored it. Those failures require different fixes.

Operational tracing can connect a request with retrieved source identifiers, versions, timing, and the resulting answer. Record only the content needed for diagnosis, with appropriate access and retention controls.

RAG is useful when it makes the evidence behind an answer accessible and relevant. Its success depends on maintaining that connection through retrieval, interpretation, and presentation.

A useful RAG answer preserves a traceable connection between the question, the evidence, and the claims it makes.