AI cost optimization: get a grip on your LLM bill before it surprises you

What starts as a proof of concept costing a few tens of euros can grow within months into monthly bills of thousands of euros from OpenAI, Anthropic or Google. We help engineering and finance teams make those LLM costs transparent, reduce them technically and keep them predictable, without compromising the quality of your AI product.

Prompt caching Model routing Batch API Semantic caching Observability FinOps for AI
Request a cost scan View strategies
LLM cost / day orange = baseline green = optimized Cache hit-rate 76% hits — input tokens 90% goedkoper Routed to Haiku/4o-mini Batch API queue 50% korting · 24h SLA

Why your LLM bill is growing faster than your product adoption

At first glance, the pricing of large language models looks straightforward: a price per million input tokens and a price per million output tokens. In practice, costs still get out of hand because tokens pile up in places engineers don't think to look: system prompts that are sent with every call, retrieval context that grows without limit, agentic loops that call themselves again and chat history that swells to tens of thousands of tokens.

A chatbot handling ten thousand conversations a day, with an average context of five thousand tokens and a top-tier model such as GPT-4o or Claude Sonnet as the only router, quickly exceeds a thousand euros a day. An agentic system that makes ten to twenty model calls per user action, for planning, tool selection, observation and reflection, multiplies that bill by a further factor of ten. If you never measure a baseline, you only find out what happened at the end of the month.

AI cost optimisation is not a one-off technical fix but an ongoing FinOps process for large language models: measuring, attributing costs per feature and team, optimising where the most cost is incurred and continuing to monitor as your volume grows. The right tool or technique depends on your workload: a batch pipeline calls for something different from a real-time copilot.

The seven levers that really reduce your LLM costs

Not every cost-saving tactic works for every workload. Below are the technical levers we steer on, ranked by typical impact when they fit your use case.

💾

Prompt caching

Anthropic and OpenAI support prompt caching: with Claude, a cache hit gives up to ninety per cent off input tokens, and with OpenAI roughly fifty per cent. It works best for long system prompts, static context and RAG fragments that stay the same across multiple requests. We structure your prompts so that the cacheable part comes first and make sure the TTL matches your traffic pattern.

🔀

Model routing

Most requests in a production pipeline are simple: classification, extraction, summarisation. Route those to a cheap model such as Claude Haiku, GPT-4o-mini or Gemini Flash and send only the complex reasoning tasks to Sonnet, GPT-4o or Opus. An LLM-as-router pattern, or dedicated routers such as Martian, Portkey or Eden AI, decides per request which model is the best fit.

📦

Batch API

For anything that doesn't need to be real-time (embedding pipelines, document classification, nightly enrichment, dataset cleaning), OpenAI Batch offers a fifty per cent discount with a twenty-four-hour SLA, and Anthropic offers a comparable Message Batches API. We split your workload into real-time and async layers and move everything that can be batched into batches, with cron scheduling and resume logic.

🧠

Semantic caching

In chatbots and search interfaces, many user questions match at a semantic level even when the exact wording differs. A vector cache (Redis with embeddings, GPTCache or Portkey semantic cache) matches similar queries and serves the existing answer. In support bots, this regularly saves us thirty to sixty per cent of model calls, provided the relevance threshold is set correctly.

✂

Context pruning & structured output

Long chat histories and RAG results full of irrelevant detail are a silent killer. We prune the context down to what the model really needs, using summary roll-ups, top-k re-ranking and token budgets per conversation. With JSON schemas or Anthropic tool use, we force shorter, structured output: no free-form prose where a list will do.

📚

RAG instead of long context

Stuffing a million tokens into the context window is possible, but it is expensive and slow. For knowledge questions, a well-built RAG pipeline (chunking, hybrid search, re-ranking) is almost always cheaper and more accurate than sending the whole document. We build retrieval layers with pgvector, Qdrant or Elastic and evaluate them with RAGAS-style frameworks.

🎯

Distillation & fine-tuning

For repetitive tasks with enough examples, we train a smaller model that mimics the behaviour of the large model, known as knowledge distillation. A fine-tuned Llama 3.1 8B or Mistral Small can run specific domains at a fraction of the cost, whether you work via Together AI, Fireworks or self-hosted.

📉

Quantization for on-prem

If you host models yourself, for compliance reasons or because your volume justifies it, quantization (AWQ, GPTQ, GGUF, FP8) lets you run the same model on smaller GPUs. A 70B model that would normally need two A100s fits, once quantized, on a single H100 or even on consumer GPUs. We calculate where the break-even point between API costs and self-hosting lies for your traffic.

📰

Observability & alerting

Without measurement, there is no optimisation. We roll out Helicone, Langfuse, OpenLLMetry or Vellum as a gateway for your LLM calls. Per feature, user and model, we see costs, latency, cache ratio and errors. Budget alerts and anomaly detection prevent a loop in production from costing you a thousand euros within an hour.

Our approach: from raw invoice to predictable unit cost

We work in four phases, with a concrete picture of savings and risks after each one. No long engagement without interim results: your bill starts dropping during the scan itself.

Cost audit

We connect to your provider billing and analyse where tokens are actually being consumed: per feature, per endpoint, per user. Often, twenty per cent of the calls cause eighty per cent of the costs. The audit report pinpoints the hot spots.

Quick wins

Within two to three weeks, we implement the low-hanging fruit: switching on prompt caching, shortening system prompts, moving work to batch. These are usually good for a twenty to forty per cent saving without changing the product.

Architectural changes

After that, we tackle model routing, semantic caching, RAG redesign and, where it makes sense, fine-tuning. This is where the structural savings come from. We build, A/B test and measure quality before and after, so you can be sure your output stays up to standard.

FinOps loop

We set up observability, dashboards and budget alerts and hand the operation over to your team. Monthly or quarterly, we join you in reviewing the set-up to factor in new models, lower prices and growing traffic.

Tooling we work with

A mature AI cost stack consists of three layers: a gateway layer for caching, routing and observability; an evaluation layer to keep quality in view during optimisations; and an orchestration layer for batch jobs and agent loops. We combine best-of-breed open source with provider-native features where those scale better and cost less.

The right choice depends on your stack. If you run on AWS Bedrock, we use cross-region inference profiles and provisioned throughput for fixed pricing. On Azure OpenAI, we use Provisioned Throughput Units (PTUs) where volume justifies it. With direct Anthropic or OpenAI keys, we lean more on gateways such as Portkey or LiteLLM. On-prem or in your own VPC, we work with vLLM, TGI or Ollama behind an internal gateway.

Helicone Langfuse OpenLLMetry Vellum Portkey LiteLLM Martian Eden AI GPTCache Redis Vector pgvector Qdrant vLLM Together AI Fireworks AWS Bedrock Azure OpenAI Anthropic Batch OpenAI Batch RAGAS
Not sure about a large project yet?

Test your idea first: a working prototype in 1 day

With OneDayBuild, we make your idea tangible in a single day for €1,150, so you know whether further development is worth the investment. Decide to go ahead with the full build? Then we deduct the cost in full.

View OneDayBuild →

Three typical workloads and what you can save on them

The right optimisation depends on how your application behaves. Below are three scenarios we often come across, along with the tactics that deliver the most for each type.

💬

Chatbot / customer support copilot

Many users, repeating intents and long system prompts with brand guidelines. The biggest levers are prompt caching for the system part, semantic caching for the most frequently asked questions, a cheap default model that escalates to a more expensive one based on intent classification, and tight chat history pruning. Savings of thirty to sixty percent are often achievable.

📄

Document pipeline / batch extraction

Processing tens of thousands of invoices, contracts or emails a day. The batch API gives you a fifty percent discount straight away; embedding models for pre-classification keep expensive LLM calls away from trivial documents; structured output via JSON schema prevents rework. For high-volume tasks, a fine-tuned smaller model is often the final step.

🤖

Agent / multi-step orchestrator

Ten to thirty model calls per user action for planning, tool selection and reflection. Per-step model routing works very well here: a cheap model for planning and tool selection, a more expensive one for the final reasoning. Caching on tool descriptions and strict loop limits with budget alerts prevent runaway costs.

What an audit looks like

An example set-up for the first week. We map every feature to costs, routing and optimisation potential, concretely and with figures.

FeatureCurrent modelProposed optimisationExpected impact
Support chatbotClaude Sonnet on every callPrompt caching + Haiku for classificationInput tokens 90% cheaper, 70% of routes to Haiku
Nightly document taggingGPT-4o real-timeOpenAI Batch + GPT-4o-mini50% batch discount + 15x lower per-token price
RAG knowledge base200K context per queryHybrid search + top-8 re-rankingContext reduced from 200K to an average of 4K tokens
Agent planner loopSonnet for every stepHaiku for planning, Sonnet for synthesis~60% lower cost per agent trace
Embedding pipelinetext-embedding-3-large3-small + dimension reduction to 1024~60% lower embedding costs

The figures in the table are typical orders of magnitude based on providers' public pricing (Anthropic, OpenAI). Your exact savings depend on traffic patterns, quality requirements and latency budgets, which we determine together during the audit phase.

Why Appfront for AI cost optimisation

We are a product development team, not a pure consulting firm. Our recommendations can always be delivered in code, not just in slides.

Engineering firstWe build the optimisations in ourselves if you want us to (gateways, caching layer, routers, observability), rather than throwing a report over the fence.
Provider-independentWe work with Anthropic, OpenAI, Google, Mistral, Cohere, AWS Bedrock and Azure OpenAI. Lock-in is not the goal; the right mix is.
FinOps for AIWe bring the tagging, attribution and alerting discipline of classic cloud FinOps to your LLM stack, including showback per team or feature.
Quality gatesOptimisations that lower the quality of your output are not optimisations. We measure with evals and regression suites before and after every change.
EU hosting where neededFor clients with GDPR requirements, we route via EU regions of Bedrock, Azure or self-hosted vLLM clusters. Compliance takes priority over any saving.
Concrete unit economicsWe don't report "costs down" but cost per conversation, per ticket, per processed document — so you can justify your product pricing.

Frequently asked questions about AI cost optimisation

How much can I realistically save on my LLM costs?
For most teams that have never optimised in a structured way, savings of thirty to seventy per cent are within reach, depending on the workload. Quick wins such as prompt caching and model routing usually deliver twenty to forty per cent within two to four weeks. Architectural changes (RAG redesign, distillation, batch migration) then bring another big step. We determine the actual room for savings during the audit.
Does prompt caching work for my use case?
Prompt caching pays off most when you have long, reusable context — an extensive system prompt, fixed tool definitions or static RAG fragments — that stays the same across many requests. For chatbots, copilots and agent systems it is almost always valuable. For one-off, one-shot calls with constantly new content it is far less so. Anthropic offers up to ninety per cent off cache hits for input tokens, OpenAI roughly fifty per cent. In the audit, we measure how much cacheable content you have and which TTL strategy fits.
What exactly is model routing and why is it so effective?
Model routing means that not every request goes to your most expensive model. Simple classification or intent detection is done just as well by Claude Haiku, GPT-4o-mini or Gemini Flash, at a fraction of the price. We only send complex reasoning or synthesis tasks to Sonnet, GPT-4o or Opus. We build routing as rule-based (on intent, length, user type) or via an LLM-as-router pattern where a cheap model makes the decision itself. Tools such as Martian, Portkey and Eden AI offer this as a managed service.
When are OpenAI Batch or Anthropic Message Batches suitable?
For anything that doesn't need to happen within seconds. OpenAI Batch gives a fifty per cent discount with a twenty-four-hour SLA, and Anthropic is comparable. Typical use cases: nightly document tagging, embedding runs, dataset cleaning, email classification, product data enrichment. We build a wrapper that splits jobs, handles retries and delivers the results to your data warehouse or database.
When does fine-tuning or your own open-source model pay off?
Fine-tuning pays off once a specific task has high volume, enough training data is available and the variation between requests is limited. For classification, extraction or style-conforming writing tasks, a fine-tuned Llama 3.1 8B or Mistral Small via Together AI or Fireworks can often run an order of magnitude cheaper than a frontier API. Self-hosting with vLLM on your own GPUs comes into play at very high volumes or strict compliance requirements — we calculate the break-even point for that.
What is semantic caching and how does it differ from prompt caching?
Prompt caching works on an exact token-prefix match and sits with the provider. Semantic caching works on your side and matches on meaning: two differently phrased but similar questions get the same cached answer. We do this via a vector store with a similarity threshold (cosine similarity). It works especially well for support bots and knowledge FAQs. We tune the threshold and build invalidation rules so that outdated answers don't linger.
How do you measure that quality doesn't drop after optimisation?
For every change, we set up evals: a fixed set of representative inputs with expected outcomes, plus an LLM-as-judge or human spot-check on subjective criteria. We run that suite before and after every change and reject the change if a key score drops. We use frameworks such as RAGAS, Promptfoo and Langfuse evals as standard. Quality always comes before cost: a cheap chatbot that hallucinates costs you more in support tickets than it saves.
Does this also work with GDPR-sensitive data and EU hosting?
Yes. We route via the EU regions of AWS Bedrock, Azure OpenAI or Google Vertex, or we self-host with vLLM, TGI or Ollama in an EU cluster. For clients in finance, healthcare or government, we build the gateway layer in your own VPC, with logs that never leave the European mainland. Optimisations such as caching, routing and distillation work at least as well in a self-hosted setup, and often better, because you have more control over the stack.

Want to get a grip on your LLM costs?

Send us your provider billing or a week of logs. We'll produce an initial analysis and show you where the biggest savings lie, with no obligation.

Book a cost scan

Edit content