Why Enterprise RAG Fails Before Retrieval: The Data Quality Problem
Enterprise RAG systems fail before retrieval because raw documents enter the ingestion pipeline with unpruned historical drafts, severed contractual clauses, and collapsed table structures. Vector databases match mathematical similarity, not operational validity or document age. When uncurated enterprise data is embedded without temporal filtering, parent-child context preservation, or layout-aware parsing, the vector index surfaces mathematically close noise, forcing the LLM to hallucinate or quote obsolete policies.
The incident usually starts with a support ticket nobody saw coming.
An enterprise lender deployed an internal RAG assistant to help loan officers navigate commercial underwriting policies. In testing, the system looked flawless: low latency, crisp answers, and leadership celebrated another AI win.
Three weeks into production, a loan officer asked for the debt-service coverage ratio (DSCR) floor on retail properties. The assistant returned a clean response: 1.15x.
The actual floor had been raised to 1.25x eighteen months earlier.
The underwriter nearly approved a high-risk credit facility on ghost policy text. In the post-mortem, the team blamed the vector stack. They debated switching from cosine distance to dot product, tuning the 512-token chunk boundaries, and swapping embedding models.
None of that mattered. When an engineer finally inspected the raw source file in the corporate drive, they found the culprit: an unpruned draft titled Underwriting_Guidelines_US_Retail_v4_Draft_Final_OLD.pdf that had been ingested on day one.
The retrieval engine didn’t break. It found the closest semantic match to the prompt and delivered it with mathematical precision. The pipeline was compromised the moment raw enterprise debt hit the parser.
The Realities of Uncurated Data
Enterprise RAG looks trivial on a whiteboard: pull docs from SharePoint or Confluence, chunk them, embed them into pg vector, and pass the top matches to an LLM.
In reality, vector search has no concept of truth or time. It measures semantic distance. When unmanaged corporate knowledge hits standard ingestion pipelines, three structural failures reliably poison retrieval before a query executes.
1. Temporal Blindness
Vector databases live in an eternal present. If your repositories contain a 2021 expense policy (75perdiem)andanupdated2025revision(100 per diem), a dense vector search treats both as equally authoritative. If the 2021 text matches the employee’s exact phrasing better than the newer policy, the older text wins the cosine match. Without explicit temporal filters, vector indexes continuously serve retired business logic as fresh ground truth.
2. Severed Context from Blind Token Slicing
Splitting text at arbitrary 500-token intervals destroys legal and operational logic. A contract may state broad liability terms in paragraph one, follow with pages of calculations, and conclude with an exclusion: "Provided, however, that penalties do not apply to scheduled maintenance announced 48 hours prior."
A token-bound chunker routinely slices the exception from the main clause. The operational exception lands in Chunk B without its governing rule in Chunk A. When an operator queries maintenance downtime, the retriever feeds an isolated caveat into the prompt, forcing the LLM to guess or hallucinate the governing terms.
3. Table Flattening
Standard PDF parsers read across rows and strip layout coordinates. When a five-column financial balance sheet is flattened into a single text string, numbers detach from their fiscal years and column headers. The resulting embedding represents an indistinct soup of financial vocabulary. When asked for Q1 margins, the model finds four unmoored percentages in the chunk and guesses.
How to Fix the Pipeline Before Ingestion
Fixing production RAG does not require a bigger foundation model. It requires disciplined data engineering applied before embedding:
- Decouple Search from Context (Parent-Child Indexing): Stop feeding your exact search chunk to the LLM. Vectorize compact chunks (100–150 tokens) for high-precision retrieval, but map them to broader parent sections (800–1,200 tokens) in a document store. Retrieve on the child vector; inject the parent block into the context window.
- Layout-Aware Extraction: Replace basic text strippers with layout-aware parsers that preserve Markdown tables, callouts, and header trees. Numbers must remain bound to their row and column keys before tokenization.
- Deterministic Metadata Gates: Enforce strict metadata validation at the boundary. Every document must carry machine-readable tags: effective_date, status (Approved/Draft), and access_group. Apply deterministic SQL-style pre-filters before vector matching runs (status='Approved' and effective_dateCurrentDate).
Grounded Retrieval Starts at the Source
RAG is an amplifier, not an automated cleaner for corporate data debt. If your internal documentation is a patchwork of superseded drafts, orphaned clauses, and collapsed tables, modern reasoning models will only help you generate bad decisions faster.
When an enterprise assistant hallucinates in production, resist the urge to immediately rewrite prompts or swap vector databases. Go look at the source files first. In almost every post-mortem, the failure was written into the text years before the AI ever parsed it.