A retrieval-augmented generation demo is one of the easiest impressive things to build in software right now. Point a model at a folder of documents, chunk them, embed them, and by the afternoon you have a chatbot answering questions about your policies. It looks finished.
It is not finished, and the gap between that afternoon and a system people can rely on is where nearly all the engineering lives. Here are the six ways RAG systems break in production, and what actually fixes each one.
1. Retrieval fails, so the model invents
The single biggest cause of a wrong answer is not the model. It is that the right chunk never made it into the context, and the model, asked a direct question with no relevant material, produced something plausible anyway.
This is a retrieval quality problem, not a prompt problem, and the fix is measurement. Build a small evaluation set of real questions with known correct source passages, then measure whether retrieval actually surfaces them. Once you can see recall, hybrid search, keyword plus vector, usually beats pure semantic search on real corpora, because people search for invoice numbers, product SKUs, and error codes that embeddings handle badly.
The other half of the fix is instruction: the system must be able to say it does not know. A model told to answer from context only, and given nothing relevant, should return “I could not find this in the documents” and a link to search. That answer is a success, not a failure.
2. Chunking destroys the meaning
Splitting documents every 500 characters is the default in every tutorial and it is wrong for most real documents. It cuts tables in half, orphans a clause from the heading that scopes it, and separates “the following exclusions apply” from the exclusions.
Chunk on structure instead: sections, headings, table boundaries, list groups. Keep the heading trail attached to each chunk so a retrieved fragment still knows which policy and which section it came from. On contracts and policy documents this one change is often worth more accuracy than swapping to a bigger model.
3. Nobody notices the documents went stale
The demo runs on a snapshot. Production runs on documents that change. Six months in, the system is confidently quoting the superseded version of a policy, and it will keep doing that indefinitely, because a stale answer looks exactly like a correct one.
Production RAG needs an ingestion pipeline with the boring properties: change detection, re-indexing, deletion that actually removes a document from the index, and a visible last-updated timestamp on every answer so a human can tell how fresh the source is.
4. Everyone can see everything
This is the failure that becomes an incident report. If retrieval runs across one index with no permission model, an HR document indexed for the HR team is one well-phrased question away from any employee who asks. The model has no concept of who is asking.
Permissions have to be enforced at retrieval, filtering the candidate set by the requesting user’s access before anything reaches the model. Not in the prompt. A prompt instruction not to reveal restricted material is a suggestion, and it does not survive a determined user.
5. There is no way to tell if a change made it better
Teams tune prompts for weeks with no way to know whether Tuesday’s version is better than Monday’s, because the only test is asking it a few questions and forming an impression. That is not a feedback loop, and without one, a system drifts.
The fix is unglamorous: a regression set of question and expected-answer pairs, run automatically on every change, scoring both retrieval and the final answer. A hundred examples is enough to catch the changes that make things worse. This is the piece most often missing when we are asked to rescue an AI project someone else started, and adding it usually reveals that the accuracy problem was in retrieval all along.
6. Costs and latency scale in ways nobody modeled
Stuffing twenty chunks into context on every request works fine at demo volume and becomes a real bill at a thousand users. Meanwhile the answer takes eleven seconds and people stop using it.
The levers are ordinary engineering: retrieve broadly then re-rank down to the few chunks that matter, cache aggressively on repeated questions, stream the response so time-to-first-token is short even when the full answer is not, and route simple queries to a cheaper model. Measure cost per answered question from week one so you find out before your users do.
Where RAG is the wrong tool entirely
Sometimes the honest answer is that you do not need retrieval. If the corpus is small and stable, put it in the prompt. If users always want the same five answers, that is a search page with good content, not an LLM. If the question is really “what is the total for Q3”, that is a database query, and a language model is a slow, expensive, occasionally wrong way to run one.
We tell clients this on the scope call, because roughly a third of the AI inquiries that reach us ship better as ordinary software. When retrieval genuinely is the right answer, though, the difference between a demo and a system is these six things, and they are most of the build.
If you have documents your team spends real hours searching, see what our AI development work involves, or get an estimate.