Production RAG Implementation: How I Build Retrieval Systems at Million-Document Scale

Every RAG demo works. Ten PDFs, one embedding model, a vector search call, and the answers look great in the walkthrough video. Then a client says: "Great — now point it at our real document store. All 1.2 million files, across a dozen departments, updated daily." That's the moment most RAG projects quietly fall apart.

I build the systems that survive that moment. This isn't a tutorial — I've already written those (linked below). This is a straight description of what I actually build and deliver for clients who need retrieval that works at real scale, and how you can get one built for you.

Why Demos Don't Survive Contact With Production

At a few hundred documents, almost any approach looks fine. Brute-force similarity search is fast enough, a single chunking rule is good enough, and nobody notices the edge cases. None of that holds at scale.

Past tens of thousands of documents, naive setups start failing in specific, predictable ways: retrieval gets noisy and starts surfacing near-misses instead of the right chunk, latency climbs, one customer's documents leak into another's answers, and quality quietly degrades because nobody built a way to measure it. These aren't exotic failures — they're the default outcome of shipping a prototype architecture as-is.

If you want the deep-dive on chunking strategy, retrieval mechanics, and eval frameworks, read Building the Perfect RAG — it covers the theory in full. This post skips the theory and shows you the resulting production system.

What I've Built: A Multi-Tenant RAG Platform Past 1 Million Documents

I've designed and shipped a multi-tenant RAG platform indexing more than one million documents, with retrieval that stays fast at that scale — not just on a good day, but as the baseline. The core architecture:

• Qdrant for dense vector search
• OpenSearch running BM25 for sparse, lexical search
• Hybrid retrieval that combines both, so results aren't dependent on either method's blind spots
• Reranking on top of the combined candidate set
• Query rewriting in front of retrieval, to clean up how real users actually phrase questions

Here's why each of those pieces earns its place.

Why Hybrid Search, Not Just Vector Search

Dense vector search is excellent at meaning — it finds the chunk that's conceptually related to a question even if the wording is completely different. It's weak, though, at exact terms: a contract clause number, a product SKU, a specific name. Sparse lexical search (BM25) is the mirror image — strong on exact matches, blind to meaning. Run both and combine the results, and you cover each other's weak spots instead of picking one and living with its gaps.

Hybrid doesn't have to mean two engines, either. I've also built the entire hybrid approach inside Qdrant alone — it stores dense and sparse vectors side by side, so a single engine handles both semantic and lexical search without a separate search cluster to operate. Fewer moving parts, same coverage. Which shape fits depends on your infrastructure, and I've shipped both.

Why Adaptive Chunking

Treating every document with the same chunking rule is one of the most common shortcuts I see — and one of the most expensive. A 40-page equipment manual, a 2-page contract, and a dense quarterly report don't share a structure, so splitting them all with the same fixed-size rule throws away the structure that made each one readable in the first place. I build chunking that adapts to document type — respecting how contracts, reports, and manuals are actually organized — rather than forcing every document through the same generic splitter.

Why Reranking and Query Rewriting

Initial retrieval is intentionally a wide net — it pulls a broad candidate set from both the dense and sparse indexes. Reranking then narrows that set with a model that looks at the query and each candidate together, which is far more accurate than similarity scores computed independently. Query rewriting sits in front of all of it, because real user queries are messy, vague, or missing context an LLM needs to retrieve well — cleaning that up before retrieval runs measurably improves what comes back.

What "Production-Grade" Actually Means

A lot of RAG systems are "production" only in the sense that they're deployed somewhere. Production-grade means something more specific:

• Multi-tenancy isolation — each customer's documents are isolated by design, so scale never becomes a data-leakage risk
• Continuous evaluation — retrieval and generation quality are measured on an ongoing basis, not eyeballed once at launch and forgotten
• Speed that holds at scale — retrieval latency stays low whether the index has ten thousand documents or a few million
• Observability — you can see why the system retrieved what it retrieved, not just what it returned
• Document-type flexibility — contracts, manuals, reports, policies, transcripts — the pipeline adapts instead of assuming one format

How I Keep a RAG System Honest: Evaluation

"Continuous evaluation" was one bullet in the list above, but it deserves its own section, because it's the practice that separates systems that stay good from systems that quietly rot. Eyeballing ten answers at launch is not evaluation. A million-document system changes constantly — new documents, new query patterns, model upgrades, chunking tweaks — and every one of those changes can make answers worse without anyone noticing until a client does.

What I actually set up looks like this:

• A golden dataset — a set of real questions with known correct answers and known source documents, grown continuously from logged production failures rather than invented in a workshop. Every pipeline change runs against it before it ships.
• Retrieval and generation measured separately — if the right chunk never came back, no amount of prompt tuning will fix the answer. Retrieval gets its own metrics (did the correct document land in the top-k, and how high?) before generation quality is even looked at.
• Evals wired into CI — a chunking change that silently hurts recall should fail the build, not surface as a support ticket three weeks later.
• Tracing in production — every query's full path (rewrite, retrieval, rerank, generation) is recorded, so when an answer is wrong you can see exactly which stage failed instead of guessing.

The Tools I Reach For

The ecosystem has largely settled on a handful of tools, and the same vendor-neutral rule applies as with vector databases — the loop matters more than the logo:

• Ragas — the standard open-source framework for RAG-specific metrics: faithfulness (is the answer grounded in the retrieved context?), answer relevancy, context precision, and context recall. Its strength is scoring without hand-labelled ground truth for every question, which makes it practical for regression checks at scale.
• LangSmith — the tracing and evaluation platform from the LangChain team, and the most polished managed option. Traces every step of every request, builds eval datasets straight from real production traffic, and runs experiments against them. If your stack is LangChain-based it's the path of least resistance, but it works fine without LangChain too.
• Langfuse — the open-source, self-hostable alternative to LangSmith. Same core idea — tracing, datasets, evals, prompt management — but it can run entirely inside your infrastructure, which is the deciding factor for clients with data-residency or compliance constraints.
• Arize Phoenix — open-source tracing and evals built on OpenTelemetry, strong if you want RAG observability folded into monitoring standards your platform team already runs.
• DeepEval and promptfoo — the CI-native end of the spectrum. DeepEval gives you pytest-style unit tests for LLM outputs; promptfoo is config-driven and slots neatly into a build pipeline. Both are the easiest way to make "the evals must pass" a merge requirement rather than a good intention.

Which of these lands in your stack depends on what you already run and where your data is allowed to live — I've set up managed and self-hosted variants both. The non-negotiable part isn't the tool. It's that evaluation runs continuously, on real traffic, with failures feeding back into the golden dataset. The full metric-by-metric breakdown — RAGAS dimensions, LLM-as-judge rubrics, retrieval metrics like MRR and NDCG — is in Building the Perfect RAG if you want the theory behind the loop.

What I Can Build for You

I've delivered systems in this same category — hybrid retrieval, adaptive chunking, production hardening — for multiple clients, in both Java and Python, across different document types and industries. And none of it is tied to one vendor: Qdrant is what the platform above runs on, but the same retrieval design works on Weaviate, Elasticsearch, or any other vector database — the right engine is whichever one fits your infrastructure and your team. If you already have a stack, I build inside it rather than pushing a rewrite you didn't ask for. If you're starting from a shaky prototype or nothing at all, the deliverable is the same class of system described above, sized to your actual document volume and use case.

If you're still deciding between RAG and fine-tuning for your use case, Why RAG beats fine-tuning for most use cases lays out when each one actually makes sense — RAG wins for the large majority of document-grounded use cases, which is exactly why it's what I build most.

Let's Build Yours

If you have a real document corpus — thousands or millions of files — and you need retrieval that actually holds up in production, get in touch through the contact form. If you'd rather see this class of system in action before we talk, the live RAG project demos are one click away. You can also see the full scope of what this kind of engagement looks like on the AI Integration services page.

Related Posts

• Building the Perfect RAG — The practical, technical deep-dive on chunking, retrieval, advanced techniques, and evals.
• Why RAG beats fine-tuning for most use cases — The strategic case for retrieval over retraining, and when fine-tuning actually wins instead.
• Agent Reliability Blueprint: SLOs, Guardrails, and Human Override — The reliability layer a RAG system needs once it's embedded inside a larger agent.
• Building a production LLM pipeline in 2025 — Broader lessons on taking LLM features from demo to production.