Retrieval-augmented generation gets pitched as a solved problem: connect a vector database to a language model, and it can now answer questions about your documents. In a demo, that's close to true. In production, across dozens of deployments spanning healthcare records, engineering documentation, and regulatory filings, the gap between a working demo and a trustworthy system lives almost entirely in decisions that have nothing to do with which model you picked.
Retrieval quality beats model size
Swapping in a larger, more expensive model rarely fixes a RAG system that's giving wrong answers. In our post-mortems, the root cause is retrieval far more often than generation: the right passage was never handed to the model in the first place, so no amount of model quality could have produced a correct answer. Before upgrading a model, we always ask a simpler question first: if a human had to answer this using only the retrieved passages, could they? If not, that's a retrieval problem, not a generation problem.
Chunking is a design decision, not a default
The default chunking strategy in most tutorials — fixed-size windows with some overlap — treats every document like undifferentiated text. It isn't. A table, a numbered procedure, and a paragraph of prose all break differently, and a chunk boundary that splits a table row from its header silently destroys the information in it. Production systems that hold up well tend to chunk by structure first (sections, tables, list items) and fall back to size-based splitting only within a structural unit, plus they preserve enough surrounding context — a heading, a document title — in each chunk so it makes sense in isolation.
Build an evaluation harness before you need one
Almost every team that skips a formal evaluation set ends up building one later anyway, under worse conditions — after a user has already been given a wrong answer. A minimal harness doesn't need to be elaborate: a few dozen real questions with known-correct answers and known-relevant source passages, checked automatically after every change to chunking, retrieval, or prompting. The value isn't statistical rigor at that scale — it's catching the regression where a "small" prompt tweak silently drops accuracy on a category of question nobody was watching.
The latency budget nobody plans for
A RAG pipeline is a chain: embed the query, search the index, rerank if you're doing it well, assemble the context, then generate. Each step adds latency, and teams that budget for "the model's response time" alone are routinely surprised when the full pipeline is three or four times slower in production than in testing. We ask clients to set a latency budget per stage before building, not after users start complaining — it changes real decisions, like whether reranking is worth its cost for a given use case, or whether a smaller retrieval set with tighter relevance beats a larger one with a slower rerank step.
Citations aren't a nice-to-have
Every production RAG system we've shipped in a regulated environment includes visible citations back to source passages — not as a trust signal for marketing, but as a debugging tool for the team and a verification tool for the user. When an answer looks wrong, the fastest way to find out why is to check whether the citation actually supports it. Systems without this are much harder to improve, because "the model was wrong" and "the retrieval was wrong" look identical from the outside.
What we'd tell a team starting today
Get a small, ugly, end-to-end version running against real documents in week one — not a polished demo, a rough pipeline you can poke at. Most of what will actually go wrong in your specific domain only shows up once real documents, real questions, and real edge cases hit the system, and no amount of upfront model selection substitutes for that early contact with reality.