Custom RAG implementation: retrieval-augmented generation for enterprise AI

When RAG, when fine-tuning, when prompt engineering alone

The three methods for giving an LLM domain knowledge have different cost structures, latency profiles and accountability characteristics. The right choice depends on how often your data changes, the volume of ground truth and your requirements for source attribution. RAG is the right starting point in nine out of ten enterprise cases.

Prompt engineering works as long as your knowledge fits within a few thousand tokens and rarely changes. Once your knowledge source exceeds the context window or changes daily (product catalogue, ticket history, legal updates), prompt engineering breaks down. Fine-tuning is meant to teach behaviour or style, not to bake in facts. A fine-tuned model that produces an invented fact is just as hard to correct as an error in a weight matrix; you cannot selectively forget it.

RAG decouples knowledge from the model. The LLM remains a general reasoning engine, while the truth lives in an external document store with version history, audit trail and access control. Updating a document is an upsert to the vector DB, not a twelve-hour retraining run. Withdrawing a document (GDPR, confidentiality obligations, an expired contract) is a delete; with fine-tuning that is effectively impossible. For enterprise applications where accountability, retractability and source attribution matter, RAG is the structurally correct architecture.

Chunking strategy, embeddings and vector database: three choices that determine everything

The three technical decisions that most directly affect retrieval quality are: how you split documents into pieces, which embedding model you choose, and which vector store you index in. There is no standard advice — the right choice depends on document length, query style, language and latency budget.

Chunking. Fixed-size chunking (for example 512 tokens with a 64-token overlap) is robust and inexpensive, but it often breaks mid-argument. Semantic chunking uses a sentence encoder to find logical break points, so chunk boundaries fall where the subject actually changes. For long legal or technical documents, hierarchical chunking works well: small chunks for retrieval precision, linked to larger parent chunks that are passed as context during synthesis. The recall score on your own golden set is the only reliable way to determine which strategy wins.

Embeddings. OpenAI text-embedding-3-large is a strong default for English and reasonable Dutch. Voyage-3 and voyage-multilingual consistently perform well on multilingual benchmarks. Cohere embed-v3 supports input-type conditioning (search_document vs. search_query), which benefits asymmetric retrieval. For self-hosted scenarios, BGE-large, e5-mistral and gte-large work well; they can be fine-tuned on your own domain if out-of-the-box accuracy falls short. Important: switching embedding model means rebuilding your entire index, so benchmark beforehand on your own data.

Vector database. Pinecone is managed, scales easily and has strong hybrid search, but cannot be self-hosted. Weaviate offers rich filtering, a GraphQL API, and self-hosted or cloud deployment. Qdrant is high-performance, Rust-based, with strong filtering and a good self-hosting story. pgvector adds vector search to an existing PostgreSQL database, which is handy if your metadata is already relational and you want to manage a single database. Milvus is built for enterprise scale (billions of vectors) but carries more operational complexity. The choice depends on scale, hosting requirements (EU-only?), filter complexity, and whether you want standalone infrastructure or an integrated stack.

Evaluation: RAGAS, golden sets and what it really means for your RAG to work

A RAG system that works in a demo says nothing about whether it will work in production. You need quantitative evaluation; otherwise every change is a guess, and you have no way of knowing whether a new embedding version is an improvement or a regression.

We deliver every RAG implementation with an eval harness that you can run yourself. A new LLM version, a different embedding model, an adjusted chunking strategy: you compare before and after, and you decide on figures rather than gut feeling. Frameworks we use as standard: RAGAS, LlamaIndex Eval, custom pytest harnesses for specific claim types, and Langfuse for production tracing.

Concrete RAG applications we build

RAG is not a single product; it is a pattern that fits almost any knowledge-intensive process. A few recurring applications where the return on investment is directly measurable.