AI cost optimization: get a grip on your LLM bill before it surprises you
What starts as a proof of concept costing a few tens of euros can grow within months into monthly bills of thousands of euros from OpenAI, Anthropic or Google. We help engineering and finance teams make those LLM costs transparent, reduce them technically and keep them predictable, without compromising the quality of your AI product.
Request a cost scan View strategiesWhy your LLM bill is growing faster than your product adoption
At first glance, the pricing of large language models looks straightforward: a price per million input tokens and a price per million output tokens. In practice, costs still get out of hand because tokens pile up in places engineers don't think to look: system prompts that are sent with every call, retrieval context that grows without limit, agentic loops that call themselves again and chat history that swells to tens of thousands of tokens.
A chatbot handling ten thousand conversations a day, with an average context of five thousand tokens and a top-tier model such as GPT-4o or Claude Sonnet as the only router, quickly exceeds a thousand euros a day. An agentic system that makes ten to twenty model calls per user action, for planning, tool selection, observation and reflection, multiplies that bill by a further factor of ten. If you never measure a baseline, you only find out what happened at the end of the month.
AI cost optimisation is not a one-off technical fix but an ongoing FinOps process for large language models: measuring, attributing costs per feature and team, optimising where the most cost is incurred and continuing to monitor as your volume grows. The right tool or technique depends on your workload: a batch pipeline calls for something different from a real-time copilot.
The seven levers that really reduce your LLM costs
Not every cost-saving tactic works for every workload. Below are the technical levers we steer on, ranked by typical impact when they fit your use case.
Prompt caching
Anthropic and OpenAI support prompt caching: with Claude, a cache hit gives up to ninety per cent off input tokens, and with OpenAI roughly fifty per cent. It works best for long system prompts, static context and RAG fragments that stay the same across multiple requests. We structure your prompts so that the cacheable part comes first and make sure the TTL matches your traffic pattern.
Model routing
Most requests in a production pipeline are simple: classification, extraction, summarisation. Route those to a cheap model such as Claude Haiku, GPT-4o-mini or Gemini Flash and send only the complex reasoning tasks to Sonnet, GPT-4o or Opus. An LLM-as-router pattern, or dedicated routers such as Martian, Portkey or Eden AI, decides per request which model is the best fit.
Batch API
For anything that doesn't need to be real-time (embedding pipelines, document classification, nightly enrichment, dataset cleaning), OpenAI Batch offers a fifty per cent discount with a twenty-four-hour SLA, and Anthropic offers a comparable Message Batches API. We split your workload into real-time and async layers and move everything that can be batched into batches, with cron scheduling and resume logic.
Semantic caching
In chatbots and search interfaces, many user questions match at a semantic level even when the exact wording differs. A vector cache (Redis with embeddings, GPTCache or Portkey semantic cache) matches similar queries and serves the existing answer. In support bots, this regularly saves us thirty to sixty per cent of model calls, provided the relevance threshold is set correctly.
Context pruning & structured output
Long chat histories and RAG results full of irrelevant detail are a silent killer. We prune the context down to what the model really needs, using summary roll-ups, top-k re-ranking and token budgets per conversation. With JSON schemas or Anthropic tool use, we force shorter, structured output: no free-form prose where a list will do.
RAG instead of long context
Stuffing a million tokens into the context window is possible, but it is expensive and slow. For knowledge questions, a well-built RAG pipeline (chunking, hybrid search, re-ranking) is almost always cheaper and more accurate than sending the whole document. We build retrieval layers with pgvector, Qdrant or Elastic and evaluate them with RAGAS-style frameworks.
Distillation & fine-tuning
For repetitive tasks with enough examples, we train a smaller model that mimics the behaviour of the large model, known as knowledge distillation. A fine-tuned Llama 3.1 8B or Mistral Small can run specific domains at a fraction of the cost, whether you work via Together AI, Fireworks or self-hosted.
Quantization for on-prem
If you host models yourself, for compliance reasons or because your volume justifies it, quantization (AWQ, GPTQ, GGUF, FP8) lets you run the same model on smaller GPUs. A 70B model that would normally need two A100s fits, once quantized, on a single H100 or even on consumer GPUs. We calculate where the break-even point between API costs and self-hosting lies for your traffic.
Observability & alerting
Without measurement, there is no optimisation. We roll out Helicone, Langfuse, OpenLLMetry or Vellum as a gateway for your LLM calls. Per feature, user and model, we see costs, latency, cache ratio and errors. Budget alerts and anomaly detection prevent a loop in production from costing you a thousand euros within an hour.
Our approach: from raw invoice to predictable unit cost
We work in four phases, with a concrete picture of savings and risks after each one. No long engagement without interim results: your bill starts dropping during the scan itself.
Cost audit
We connect to your provider billing and analyse where tokens are actually being consumed: per feature, per endpoint, per user. Often, twenty per cent of the calls cause eighty per cent of the costs. The audit report pinpoints the hot spots.
Quick wins
Within two to three weeks, we implement the low-hanging fruit: switching on prompt caching, shortening system prompts, moving work to batch. These are usually good for a twenty to forty per cent saving without changing the product.
Architectural changes
After that, we tackle model routing, semantic caching, RAG redesign and, where it makes sense, fine-tuning. This is where the structural savings come from. We build, A/B test and measure quality before and after, so you can be sure your output stays up to standard.
FinOps loop
We set up observability, dashboards and budget alerts and hand the operation over to your team. Monthly or quarterly, we join you in reviewing the set-up to factor in new models, lower prices and growing traffic.
Tooling we work with
A mature AI cost stack consists of three layers: a gateway layer for caching, routing and observability; an evaluation layer to keep quality in view during optimisations; and an orchestration layer for batch jobs and agent loops. We combine best-of-breed open source with provider-native features where those scale better and cost less.
The right choice depends on your stack. If you run on AWS Bedrock, we use cross-region inference profiles and provisioned throughput for fixed pricing. On Azure OpenAI, we use Provisioned Throughput Units (PTUs) where volume justifies it. With direct Anthropic or OpenAI keys, we lean more on gateways such as Portkey or LiteLLM. On-prem or in your own VPC, we work with vLLM, TGI or Ollama behind an internal gateway.
Test your idea first: a working prototype in 1 day
With OneDayBuild, we make your idea tangible in a single day for €1,150, so you know whether further development is worth the investment. Decide to go ahead with the full build? Then we deduct the cost in full.
View OneDayBuild →Three typical workloads and what you can save on them
The right optimisation depends on how your application behaves. Below are three scenarios we often come across, along with the tactics that deliver the most for each type.
Chatbot / customer support copilot
Many users, repeating intents and long system prompts with brand guidelines. The biggest levers are prompt caching for the system part, semantic caching for the most frequently asked questions, a cheap default model that escalates to a more expensive one based on intent classification, and tight chat history pruning. Savings of thirty to sixty percent are often achievable.
Document pipeline / batch extraction
Processing tens of thousands of invoices, contracts or emails a day. The batch API gives you a fifty percent discount straight away; embedding models for pre-classification keep expensive LLM calls away from trivial documents; structured output via JSON schema prevents rework. For high-volume tasks, a fine-tuned smaller model is often the final step.
Agent / multi-step orchestrator
Ten to thirty model calls per user action for planning, tool selection and reflection. Per-step model routing works very well here: a cheap model for planning and tool selection, a more expensive one for the final reasoning. Caching on tool descriptions and strict loop limits with budget alerts prevent runaway costs.
What an audit looks like
An example set-up for the first week. We map every feature to costs, routing and optimisation potential, concretely and with figures.
| Feature | Current model | Proposed optimisation | Expected impact |
|---|---|---|---|
| Support chatbot | Claude Sonnet on every call | Prompt caching + Haiku for classification | Input tokens 90% cheaper, 70% of routes to Haiku |
| Nightly document tagging | GPT-4o real-time | OpenAI Batch + GPT-4o-mini | 50% batch discount + 15x lower per-token price |
| RAG knowledge base | 200K context per query | Hybrid search + top-8 re-ranking | Context reduced from 200K to an average of 4K tokens |
| Agent planner loop | Sonnet for every step | Haiku for planning, Sonnet for synthesis | ~60% lower cost per agent trace |
| Embedding pipeline | text-embedding-3-large | 3-small + dimension reduction to 1024 | ~60% lower embedding costs |
The figures in the table are typical orders of magnitude based on providers' public pricing (Anthropic, OpenAI). Your exact savings depend on traffic patterns, quality requirements and latency budgets, which we determine together during the audit phase.
Why Appfront for AI cost optimisation
We are a product development team, not a pure consulting firm. Our recommendations can always be delivered in code, not just in slides.
Frequently asked questions about AI cost optimisation
Want to get a grip on your LLM costs?
Send us your provider billing or a week of logs. We'll produce an initial analysis and show you where the biggest savings lie, with no obligation.
Book a cost scan