Putting RAG copilots in production: what actually works

Demoing a RAG copilot takes a weekend. Running one in production for a year — and watching it stay useful — is a different problem entirely. Here are the lessons we keep relearning.

1. Evals before optimizations

The first instinct is to tune chunk size, swap embedding models, and try every reranker. Don't. Build a versioned evaluation set first — even 50 honest examples from your real users.

2. Retrieval is mostly a data problem

If your retriever is failing, the issue is usually in your source documents — duplicates, stale revisions, scanned PDFs without OCR. Cleaning the corpus moves the needle more than swapping embedding models.

• Deduplicate aggressively (hash + near-dup detection).
• Tag each document with a date and an owner; expire what nobody owns.
• OCR scans before chunking. A blurry scan retrieves nothing useful.

3. Treat the system prompt like code

Version it. Diff it. Review it. The biggest behavior changes we've shipped came from prompt revisions, not model upgrades. Keep prompts in your repo, not in a notebook.

4. Guardrails are a product feature

Users notice when the assistant refuses helpful answers. They notice when it confidently makes things up. Define explicit no-go answers and explicit always-do answers.

5. Observability is non-negotiable

You need to answer three questions for any conversation: what did the user ask, what did we retrieve, and why did we say what we said. Trace IDs everywhere. Sample 100% of failures. Sample 1% of successes.

6. Costs creep in the dumbest places

Embedding every document on every deploy. Re-summarizing the same conversation history. Logging full prompts to a managed log service. Audit your spend monthly, not quarterly.

"A RAG copilot is 20% model, 80% janitorial work on your data and prompts. The teams that win at this know it."