Custom RAG implementation: retrieval-augmented generation for enterprise AI
Your LLM hallucinates, doesn't know your products and can't cite a source. Retrieval-augmented generation solves that without fine-tuning: real-time grounding on your own knowledge sources (documentation, contracts, tickets, wikis, product data), with citation grounding for accountability and hybrid retrieval for high precision. We build production RAG that measures what it knows and knows what it doesn't.
Discuss your RAG use case View architectureWhen to use RAG, when to fine-tune, when prompt engineering is enough
The three ways of giving an LLM domain knowledge have different cost structures, latency profiles and accountability properties. The right choice depends on how often your data changes, how much ground-truth data you have and your requirements for source attribution. In nine out of ten enterprise cases, RAG is the right starting point.
Prompt engineering works as long as your knowledge fits in a few thousand tokens and rarely changes. Once your knowledge source is larger than the context window or changes daily (product catalogue, ticket history, legal updates), prompt engineering breaks down. Fine-tuning is meant to teach behaviour or style, not to bake in facts. A fine-tuned model that produces a made-up fact is as hard to correct as an error in a weight matrix; you can't selectively make it forget.
RAG decouples knowledge from the model. The LLM remains a general-purpose reasoning engine, while the truth lives in an external document store with version history, an audit trail and authorisation. Updating a document is an upsert on the vector DB, not a twelve-hour retraining run. Withdrawing a document (GDPR, confidentiality obligations, an expired contract) is a delete; with fine-tuning that is effectively impossible. For enterprise applications where accountability, retractability and source attribution matter, RAG is the structurally right architecture.
RAG architecture: ingestion, retrieval, synthesis, observability
A production RAG system consists of two pipelines that run at different cadences: ingestion (batch or streaming) and retrieval (per query, sub-second). On top of these sits an observability layer that measures whether the system still does what it was built to do.
Ingestion pipeline
Document loaders for PDF, DOCX, HTML, Confluence, SharePoint, Notion, Jira, Zendesk, S3 and databases. Layout-aware parsing (Unstructured.io, LlamaParse, Azure Document Intelligence) so that tables, headers and figure captions are not lost. Then chunking (semantic or fixed with overlap), metadata enrichment (author, date, confidentiality class, source URI) and batch embedding via OpenAI text-embedding-3-large, voyage-3, Cohere embed-v3 or open-source BGE/e5. Indexing in the vector DB with a hybrid index (dense + BM25/SPLADE).
Retrieval pipeline
Query understanding (rewriting, expansion, HyDE), parallel hybrid retrieval (top-50 dense + top-50 BM25), reciprocal rank fusion, reranking down to the top 5 with Cohere Rerank or ColBERT v2, and context assembly with deduplication. The synthesis prompt explicitly asks for citations per claim. Optionally: an agentic loop with tool use for multi-hop questions where a single retrieval step isn't enough.
Observability and evaluation
Tracing via Langfuse, Helicone, LangSmith or OpenTelemetry: every retrieval, every prompt and every answer is recorded. Offline evaluation on golden sets with RAGAS (faithfulness, answer relevancy, context precision, context recall) and LlamaIndex Eval. Online evaluation through thumbs-up/down, escalation rate and human audits. Drift detection: if context recall drops or the citation-grounding score falls, an alert is triggered.
We deliberately keep these three pipelines separate. Ingestion can fail without retrieval breaking down, and retrieval can be slow without eval runs having to wait for it. Each component can be scaled, debugged and replaced independently, so swapping Pinecone for Qdrant or pgvector is a config change to a single component, not a rebuild of the system.
Chunking strategy, embeddings and vector database: three choices that determine everything
The three technical decisions with the most direct impact on retrieval quality are how you split documents into chunks, which embedding model you choose, and which vector store you index on. There is no standard answer: the right choice depends on document length, query style, language and latency budget.
Chunking. Fixed-size chunking (for example 512 tokens with 64 overlap) is robust and cheap, but often cuts off mid-argument. Semantic chunking uses a sentence encoder to find logical breakpoints, so the chunk boundary falls where the topic actually changes. For long legal or technical documents, hierarchical chunking works well: small chunks for retrieval precision, linked to larger parent chunks that are passed along as context during synthesis. The recall score on your own golden set is the only reliable way to determine which strategy wins.
Embeddings. OpenAI text-embedding-3-large is a strong default for English and reasonable Dutch. Voyage-3 and voyage-multilingual consistently score high on multilingual benchmarks. Cohere embed-v3 supports input-type conditioning (search_document vs. search_query), which benefits asymmetric retrieval. For self-hosted scenarios, BGE-large, e5-mistral and gte-large work well; they can be fine-tuned on your own domain if out-of-the-box accuracy falls short. Important: switching embedding models means rebuilding your entire index, so benchmark on your own data first.
Vector database. Pinecone is managed, scales easily and has strong hybrid search; self-hosting is not possible. Weaviate offers rich filtering, a GraphQL API and self-hosting or cloud. Qdrant is high-performance and Rust-based, with strong filtering and a good self-hosting story. pgvector adds vector search to an existing PostgreSQL database, which is handy if your metadata is already relational and you want to manage a single database. Milvus is enterprise-scale (billions of vectors), but heavier in operational complexity. The choice depends on scale, hosting requirements (EU-only?), filter complexity and whether you want separate infrastructure or an integrated stack.
Hybrid retrieval, reranking and advanced RAG patterns
Pure dense retrieval (embeddings only) fails predictably on exact terms, product codes, proper names and negations. Pure BM25 misses semantic relationships. Production RAG combines the two and uses a reranker to filter the top 50 candidates down to the top 5 that actually go into the prompt.
Hybrid retrieval (BM25 + dense)
Two parallel queries: BM25 (lexical) for exact matches and dense (semantic) for conceptual matches. Results are merged via Reciprocal Rank Fusion or a weighted combination. On most enterprise corpora this delivers a 10-25% recall improvement over dense-only retrieval โ especially on queries containing product codes, customer names and specific terms that were not in the embedding training data.
Reranking (Cohere Rerank, ColBERT v2)
A cross-encoder that calculates a relevance score for each candidate document against the query. It is far more expensive than bi-encoder retrieval, so it is only applied to a top-K. Cohere Rerank is an API call; ColBERT v2 can be self-hosted and quantised. Reranking often adds another 5-15% precision and ruthlessly filters out chunks that look semantically similar but don't actually answer your question.
HyDE and query expansion
Hypothetical Document Embeddings: you let the LLM first generate a hypothetical answer to the query, then search using the embedding of that answer. This works surprisingly well for questions that match poorly with document style (a short question against long manuals). Query expansion (synonyms, related terms) increases recall but must be done carefully โ expanding too broadly undermines your precision.
Agentic RAG and GraphRAG
For multi-hop questions ("which contracts expire in Q3 and also have a minimum-purchase clause?") a single retrieval step is not enough. Agentic RAG gives the model tools to search iteratively, filter and ask sub-questions. Microsoft GraphRAG builds an entity graph from the corpus up front, so queries about relationships between entities can be answered directly via the graph โ faster and more precise than repeated embedding searches.
Multi-modal RAG
Tables in PDFs, diagrams, screenshots and scanned contracts are out of reach for pure text embeddings. We use layout-aware parsers (Unstructured.io, Azure Document Intelligence, LlamaParse) plus multi-modal embeddings (CLIP, voyage-multimodal, Cohere embed-multimodal) to index tables as structured data and figures as visual embeddings. The result: a question about a specific row in a table lands on the right cell, not on a surrounding paragraph.
Citation grounding and attribution
The model is explicitly instructed to link every claim to a document ID and section. A citation marker appears next to each statement in the output, and the UI renders it as a clickable source. This is not a cosmetic detail โ it is what makes audit requirements, legal review and user trust possible. A RAG system without citations is not production-ready.
Evaluation: RAGAS, golden sets and what it really means for your RAG to work
A RAG system that works in the demo tells you nothing about a RAG system that works in production. You need quantitative evaluation โ otherwise every change is a gamble and you won't know whether a new embedding version is an improvement or a regression.
Faithfulness
The degree to which the answer is actually grounded in the retrieved context. A hallucinated-fact rate is the nastiest regression here: the system sounds convincing but makes things up. RAGAS scores this by checking, claim by claim, whether it can be derived from the context.
Answer relevancy
Does the answer actually fit the question? A correct but irrelevant answer ("the documentation says X") is still a failure if the user asked for something more specific. It is often measured with an LLM judge that scores answer-question pairs.
Context precision & recall
Precision: of the retrieved chunks, how many are actually relevant. Recall: of all relevant chunks in the corpus, how many were retrieved. These two metrics are the purest measure of retrieval quality, independent of LLM behaviour. NDCG and MRR are classic IR metrics that refine this further.
Golden sets and regression tests
50 to 200 question-answer pairs validated by domain experts and linked to the specific documents that contain the answer. Every deploy runs against this set. Making a regression visible is half the work; without a golden set, you are working on impressions.
We deliver every RAG implementation with an eval harness that you can run yourself. A new LLM version, a different embedding model, an adjusted chunking strategy: you compare before and after, and decide on numbers rather than gut feeling. The frameworks we use as standard: RAGAS, LlamaIndex Eval, custom pytest harnesses for specific claim types, and Langfuse for production tracing.
Test your idea first: a working prototype in 1 day
With OneDayBuild, we make your idea tangible in a single day for โฌ1,150, so you know whether further development is worth the investment. Decide to go ahead with the full build? Then we deduct the cost in full.
View OneDayBuild โHow we build RAG implementations
Iterative, with measurable deliverables per phase. No big upfront architecture that collapses in month three, but a working system in week four that you can validate yourself.
Discovery and corpus analysis
Which documents, what volume, which format, which confidentiality class. We map out use cases and define 30-50 example questions with domain experts. We build a first golden set.
Baseline RAG in 2-3 weeks
A working prototype: ingestion pipeline, hybrid retrieval and a basic UI with citation grounding. First measurement against the golden set. This is your starting point, and all follow-up work improves on it.
Iterative improvement
Adding a reranker, tuning the chunking strategy, testing an embedding model swap, query rewriting, prompt engineering on the synthesis step. Every iteration is measured on faithfulness, recall and relevancy.
Production and observability
Auth, RBAC, audit logging, tracing (Langfuse), drift monitoring and alerts on quality regressions. Documentation and knowledge transfer so your own team can run the eval harness.
The stack we use as standard
We stay model- and vendor-agnostic. The right choice depends on your hosting requirements (EU-only, on-prem, hybrid), scale, budget and latency requirements. The list below is what we have used in production, not what we sell.
For clients with strict data residency requirements (financial institutions, healthcare, government), we run the full stack on Azure West Europe or a Dutch cloud provider, with self-hosted Qdrant or pgvector and open-source LLMs via vLLM or Azure OpenAI Service in the EU region. No US data egress, no vendor lock-in on a single SaaS platform.
Concrete RAG applications we build
RAG is not a single product; it is a pattern that fits into almost any knowledge-intensive process. A few recurring applications where the ROI is directly measurable.
Internal knowledge assistant
Confluence, SharePoint, Notion, Google Drive, Slack archives, Jira tickets, support history: combined into a single chat interface where new employees and specialists find answers without phoning six people. With RBAC so everyone only sees what they are allowed to see, and citation grounding so statements link back to the source page.
Customer support copilot
An interface for your support staff that retrieves relevant knowledge base articles, previous tickets and product documentation based on the customer's question, and drafts a reply. The agent checks and sends it: average handle time goes down, quality and consistency go up. Optionally, a tier-1 bot can answer customers directly, with escalation logic.
Legal and contract Q&A
Make contracts, NDAs, terms and conditions, case law and internal legal memos searchable, with citation grounding. "Which contracts have a minimum-purchase clause and expire in Q3?" is a question that takes days to answer manually; here it is answered in seconds, with direct references to article numbers and pages.
Production and operational documentation
Work instructions, machine manuals, safety procedures and QA protocols made available to operators on the shop floor through a voice or touch interface. Multi-modal RAG that retrieves diagrams and step-by-step photo sequences. Applicable in pharma, manufacturing, logistics and healthcare.
Data governance, RBAC and GDPR with RAG
Document-level authorisation
RBAC propagates all the way down to the retrieval layer: a user only gets back chunks from documents they have access to. We link on user claims (Azure AD, Okta, custom IdP) and filter on metadata in the vector DB. No leaks via a chat interface.
GDPR compliance and data residency
Documents containing personal data are handled in line with GDPR: purpose limitation, data minimisation, retention. Where EU residency is required, the entire stack runs in an EU region. We provide a subprocessor overview and data processing agreements as standard.
Audit trail and retractability
Every answer is logged with the query, retrieved context, citations and model version. Withdrawing a document means a delete on the vector DB plus a verification run. That is something fine-tuning cannot do, and it is a key argument for RAG in regulated sectors.
Frequently asked questions about RAG implementation
Ready to build a RAG implementation?
Discuss your use case with us. We analyse your corpus, define an initial golden set and deliver a working baseline within a few weeks: measurable, with citations, production-ready.
Schedule a technical session