For twenty years, data engineering meant moving structured rows between systems. LLMs changed the raw material: the knowledge an enterprise wants its AI to use lives in PDFs, wikis, contracts, and support tickets — and retrieval-augmented generation (RAG) is how that knowledge reaches the model without retraining it.
The pattern is simple to whiteboard: parse documents, split them into chunks, embed the chunks, store the vectors, retrieve the relevant ones at question time. The gap between that whiteboard and a production system is almost entirely a data engineering problem.
Parsing is where quality is won or lost
Garbage in applies with brutal force here: a mangled table in a parsed PDF becomes a confident wrong answer at query time. Real documents have multi-column layouts, tables, headers that carry meaning, and scanned pages needing OCR. Layout-aware parsers that preserve structure — table extraction to markdown, heading hierarchies kept intact — routinely beat naive text extraction by a wide margin.
Treat parsing as a first-class pipeline stage with its own tests: golden documents whose expected output is checked on every run, exactly as you would test a dbt model.
Chunking and metadata: the underrated levers
Chunks must be small enough to embed precisely but large enough to carry context. Fixed-size splitting is a baseline, not a strategy — structure-aware chunking that respects section boundaries, with modest overlap, performs better on almost every corpus. When a chunk loses its context ('the policy above does not apply to contractors' — which policy?), prepend the document title and section heading to the chunk before embedding.
Metadata is what turns similarity search into reliable retrieval. Extracting document type, date, department, and access level alongside each chunk enables filtered queries — 'search only current HR policies' — and, critically, lets you enforce document-level permissions at retrieval time rather than hoping the model keeps secrets. A vector store without an access-control story is a data leak waiting for a clever prompt.
It's a pipeline, not a script
The demo is a notebook; the product is an idempotent, monitored pipeline. Documents change, so you need change detection and re-embedding without full rebuilds. Embedding models version, so store the model name alongside every vector — mixing embeddings from different models silently corrupts retrieval. Vector databases like Pinecone or Milvus handle the serving layer, but freshness, dedup, and lineage remain your job.
Measure retrieval like the production system it is: a golden set of questions with known source passages, recall checked on every deploy. Teams that skip this discover their AI feature degraded weeks after the fact — from a user complaint.