Retrieval-augmented generation is often introduced as a simple recipe: split documents, create embeddings, retrieve similar chunks, and place them in a model prompt. That can produce a convincing prototype. It is not yet a dependable enterprise knowledge system.

A practical RAG pipeline begins earlier, with decisions about which sources are authoritative, who may see them, how changes are detected, and what evidence an answer must provide. It ends later, with evaluation, monitoring, and a way to distinguish a retrieval failure from a generation failure.

The unit of design is the whole information path, not the vector database.

Start with source policy

Before ingestion, define the knowledge boundary:

  • Which repositories and document sets are approved?
  • Which source wins when two documents conflict?
  • How are drafts, obsolete versions, and duplicates treated?
  • Which users or groups may access each source?
  • How quickly must updates and deletions appear in the index?
  • What source information must accompany an answer?

If the source estate is uncontrolled, retrieval can make the disorder easier to query without making the answer more reliable.

Access control must also survive transformation. A restricted document should not become broadly visible because its chunks were copied into a shared index. Store source identity, version, ownership, and access metadata with every retrievable unit, then enforce those rules at query time.

Make ingestion observable

Ingestion is a software pipeline with failure modes. Files may be unreadable, tables may lose structure, scanned pages may need OCR, and a parser may silently omit content. Treat ingestion as a versioned process.

For each source, retain:

  • a stable source identifier and location;
  • a content hash or version;
  • ingestion time and parser version;
  • extraction warnings;
  • document structure and section metadata;
  • the policy used to include or exclude it.

Chunking should follow the information structure where possible. Fixed token windows are useful, but headings, paragraphs, tables, code blocks, and requirement identifiers often provide better boundaries. Overlapping chunks can preserve context, while excessive overlap can fill results with near-duplicates.

There is no universally correct chunk size. The right choice depends on how users ask questions and how evidence is expressed in the source material.

Retrieve for the question, not the demo

Semantic similarity is one retrieval signal. Exact identifiers, product codes, error messages, dates, and names may need lexical search or metadata filters. In many enterprise systems, hybrid retrieval is more useful than relying on a single mechanism.

A retrieval stage may include:

  1. query interpretation or rewriting;
  2. access and metadata filtering;
  3. semantic and lexical retrieval;
  4. result fusion;
  5. reranking;
  6. duplicate removal;
  7. context selection within a defined budget.

Log the candidates and scores before and after each step. Otherwise, a poor answer leaves no way to determine whether the required passage was unavailable, not retrieved, discarded during reranking, or lost when the context was assembled.

Assemble context deliberately

Retrieved passages are not automatically a good prompt. The context builder should preserve source identity, separate instructions from evidence, and make conflicts visible.

Useful context assembly rules include:

  • include only sources the current user may access;
  • preserve headings and local structure;
  • label each passage with a stable citation identifier;
  • prioritise authoritative and current sources;
  • avoid repeating equivalent chunks;
  • state when relevant sources disagree;
  • reserve space for the model to answer clearly.

The system should also be able to abstain. If retrieval does not provide sufficient evidence, “not enough approved information” is often the correct result.

Require grounded outputs

A grounded answer should make it possible to inspect the relationship between claim and source. That does not mean attaching a long list of vaguely related documents. Citations should point to the passages that support the material statements in the answer.

Structured outputs can help separate:

  • the answer;
  • cited source identifiers;
  • uncertainties or conflicts;
  • missing information;
  • any proposed follow-up action.

The application should validate that cited sources were actually present in the retrieved context and remain accessible to the user.

Diagnose the right failure

RAG quality problems have different causes:

  • Source failure: the required information is absent, obsolete, or contradictory.
  • Ingestion failure: the information exists but was parsed or indexed incorrectly.
  • Retrieval failure: a relevant unit exists but was not selected.
  • Context failure: the right evidence was retrieved but truncated, duplicated, or poorly arranged.
  • Generation failure: sufficient evidence was available but the response misused it.
  • Citation failure: the answer is useful, but its references are missing or misleading.

Evaluation should label these separately. Tuning the model will not repair a missing source, and changing embeddings will not repair a permission leak.

Build an evaluation set from real work

Create representative questions with expected evidence, not only expected wording. Include:

  • direct factual lookup;
  • questions requiring multiple sources;
  • exact identifiers and error strings;
  • conflicting or superseded documents;
  • questions the approved sources cannot answer;
  • access-controlled cases;
  • recently changed or deleted content.

Measure whether the necessary evidence was retrieved, whether material claims are supported, whether citations are correct, and whether the answer appropriately abstains. Human review remains valuable for relevance and usefulness, but the review should diagnose the pipeline stage responsible for the error.

Operate freshness and revocation

Enterprise knowledge changes. A production design needs scheduled or event-driven updates, deletion propagation, index versioning, and a recovery path when ingestion fails.

Monitor source coverage, ingestion lag, parse errors, empty retrieval, access-filter decisions, citation validity, latency, and user feedback. Keep the application able to identify which index and source versions contributed to a response.

A RAG system earns trust when it can explain where an answer came from, refuse when evidence is weak, and remove information when its source or permissions change. Retrieval is essential, but it is only one controlled stage in that system.

Source notes for review