AI POC development Eval-driven Path to production

AI POC development: from idea to working proof of concept

Most AI pitches end in a report. Appfront builds POCs that work on your own data, measured with an eval set, and with an architecture that scales to production without a rebuild. For CTOs, product leads and innovation teams who want to go beyond the demo.

2-6 wklead time
3 layersdata, model, eval
1 verdictproduction-ready or not
Open stackyour code, your choice

Why 70% of AI POCs never reach production

Research from Gartner and MIT shows time and again that the vast majority of enterprise AI POCs stall before reaching production. Not because AI doesn't work, but because the POC phase was set up wrongly. We see four patterns recur.

Scope creeps unnoticed

The POC started as an "answer bot for invoice questions" and ended up as "replace our entire customer service team". Model quality plummeted, and no conclusion could be drawn.

No eval set

The team looks at ten sample outputs, decides it "looks good", and stops there. At production launch, it turns out to fail dramatically on the 10,000 edge cases nobody looked at.

Data gap

The POC works beautifully on a polished test set of 50 records. Production data contains unstructured notes, legacy formats and anomalies the POC model was never trained or tested on.

Stack mismatch

The POC runs in a notebook with hard-coded keys, synchronous API calls and no logging. For production, everything has to be redone: orchestration, monitoring, security, scalability. Effectively two builds.

Three types of AI POC we build

Not every AI question calls for the same answer. By choosing the right POC type up front, we avoid discovering halfway through that the approach doesn't fit the task.

Type 1

RAG POC

Retrieval-augmented generation for questions about your own documents, policies or knowledge base. Eval focus: faithfulness, citation accuracy, retrieval recall. Typical timeline: 2-3 weeks.

  • Vector store and embeddings choice
  • Chunking strategy and retrieval tuning
  • Citation validation
  • RAGAS eval as a baseline
Type 2

Agent POC

An agent that plans, calls tools and takes action: research flows, workflow automation or multi-step tasks. Eval focus: success rate per step, hallucination rate in tool calls, cost per run. Typical timeline: 4-6 weeks.

  • Tool design (not an API mirror)
  • Plan-and-execute with human-in-the-loop
  • Trace system and run comparison
  • Approval gates for risky actions
Type 3

Classifier POC

Classification, extraction or structured output on text (emails, tickets, contracts). Eval focus: precision, recall, F1 per class. Typical timeline: 2-4 weeks, often with fine-tuning or structured-output models.

  • Golden dataset (often 300-3,000 cases)
  • Pydantic/JSON schema for output
  • Multi-model comparison
  • Production monitoring with drift detection

How Appfront builds an AI POC

A fixed rhythm in four phases. No lengthy discovery rounds; we work with scope-aware milestones so that after each phase we know whether to continue or adjust course.

Phase 01

Scope and eval design (week 1)

Together with you, we define which specific task the AI must perform, for whom, on which data, and with which success criteria. Then we design the eval set: 50 to 500 test cases covering the happy path and the most important edge cases.

  • Scope document with explicit boundaries
  • Eval set v1 (validated by your experts)
  • Definition of success per test case
Phase 02

First working version (weeks 2-3)

We build the minimal working implementation: data pipeline, model calls, output. No UI yet, no integrations: just the core. At the end of phase 2, we run the eval set and have a baseline score.

  • Working pipeline (data → model → output)
  • First eval report with baseline
  • Cost-per-run estimate
Phase 03

Iteration and improvement (weeks 3-5)

Based on the baseline, we know where it breaks. Iteration targets the bottleneck: a better prompt, a different chunking strategy, a different model, better tool design, extra context. Every iteration is measured against the same eval set, so improvement is objective.

  • Iteration reports with eval diff
  • Trace system live
  • Edge-case report
Phase 04

Production path and handover (weeks 5-6)

We build the architecture blueprint showing what production requires: orchestration, monitoring, security, cost control, fallbacks. We deliver working code, the eval suite, the trace system and the architecture document.

  • Production architecture blueprint
  • Working POC codebase and tests
  • Eval suite that can run in CI
  • Go/no-go verdict with cost impact

From POC to production without a rebuild

The biggest cost spike in AI projects comes at the transition from POC to production. With us, that transition is prepared during the POC, not afterwards.

An eval suite that fits in CI

The eval set we design in phase 1 doesn't just run on your laptop. We build it to run on GitHub Actions or GitLab CI, with output to your own dashboards. With every prompt, model or pipeline change, you know immediately whether quality got better or worse.

A separable orchestration layer

Model calls go through a thin abstraction layer that is identical in the POC and in production. Switching between Claude, GPT-4 and self-hosted is a config change, not a rebuild. For agent flows we use MCP, so tools can be reused across environments.

Observability from day 1

Logging, tracing and cost attribution are built in from the POC onwards. When rolling out to production, we don't have to bolt on monitoring infrastructure afterwards. From day 1 you can see performance and costs per user, per use case and per model.

Security and compliance documented

Data flows, prompt injection mitigations and log retention are explicitly documented in phase 4. Audit questions during the production phase can then be answered the same week, not months later after a second engagement.

Not sure about a large project yet?

Test your idea first: a working prototype in 1 day

With OneDayBuild, we make your idea tangible in a single day for €1,150, so you know whether further development is worth the investment. Decide to go ahead with the full build? Then we deduct the cost in full.

View OneDayBuild →

Example POCs from our work

Anonymised examples of POCs we have recently delivered. Including the baseline, the improvement process and the production outcome.

RAG · Finance

Policy Q&A for a compliance team

Question: can an AI answer questions about internal compliance documents with citations? Eval: 200 questions, assessed on faithfulness and citation accuracy. Baseline: 62% correct with direct citations. Result at week 4: 91% correct after chunk tuning and hybrid retrieval. Production: rolled out to 80 employees, ~400 questions per week.

Agent · Logistics

Invoice-matching agent with approval

Question: can an agent match incoming invoices to POs and involve a human when in doubt? Eval: 500 historical invoices, measured on match rate and false positives. Baseline: 71% auto-match, 12% false positives. Result at week 5: 86% auto-match, 2% false positives after a tool redesign. Production: 4 months later, saves the team ~16 hours per week.

Classifier · Healthcare

Triage of incoming patient messages

Question: can a classifier route patient emails to the right department? Eval: 1,500 labelled emails, 12 classes. Baseline: 79% accuracy with GPT-4o-mini. Result at week 3: 92% accuracy with a fine-tuned smaller model that runs 8x cheaper. Production: running for 6 months, 2,000 emails per day, error margin stable.

Common mistakes and how we avoid them

Lessons from AI POCs we have delivered for a wide range of clients, and the patterns we build in up front to avoid them.

Mistake: building a POC without an eval set
Us: the eval set is phase 1, before a single line of code is written
Mistake: only seeing production data in week 4
Us: data analysis is part of phase 1, otherwise anomalies block the whole POC
Mistake: one long demo at the end
Us: an interim result per phase, with go/no-go decision points built in
Mistake: only calculating costs after production rollout
Us: cost per run is part of the eval, visible at every iteration
Mistake: applying AI to a use case where it adds no value
Us: honest in the scoping phase: "a SQL query is better here"

Frameworks and stack we use

Pragmatic per use case. No vendor evangelism; just a list of tools we have experience with and know when they fit.

Model layer

  • Claude (Anthropic): reasoning, long context
  • GPT-4o/o1 (OpenAI): agent flows, structured output
  • GPT-4o-mini: batch work, cheap classifiers
  • Llama 3.1 / 3.3 (self-hosted): sensitive data, high volume
  • Mistral / Qwen: cost-efficient self-hosting

Tools and orchestration

  • MCP (Model Context Protocol): tool layer
  • OpenAI function calling / Structured Output
  • LangChain / LangGraph: agent orchestration
  • LlamaIndex: RAG pipelines
  • Pydantic-AI: typed agent output

Evals and observability

  • Promptfoo: prompt regression suites
  • RAGAS: RAG-specific metrics
  • Langfuse / Arize: tracing and monitoring
  • OpenAI Evals: standardised evals
  • Custom pytest runners: for specific metrics

Production layer

  • vLLM / TGI: high-throughput inference
  • Ollama: local deployment
  • Pinecone / Qdrant / Weaviate: vector stores
  • Postgres + pgvector: for existing Postgres stacks
  • Kubernetes deployments with Helm charts

Why Appfront for your AI POC

Engineering-first

We build, we evaluate, we deliver working code. No consultancy decks; just a POC that your own team can take over, or that we keep building on all the way to production.

Production experience

Our team has put AI systems into production for clients in finance, healthcare, logistics and the public sector. We know the pitfalls that only become visible once real traffic arrives.

The code stays yours

The code is yours. We work with open frameworks and standard cloud tooling where it fits. You can maintain it yourself, have someone else develop it further, or carry on with us.

An honest verdict

If the POC shows that AI isn't the right solution here, we'll tell you. That hasn't cost us clients; on the contrary, it's the reason clients come back for their next project.

Ready to put your AI POC to an honest test?

Describe your use case in a few sentences. We'll schedule a 45-minute intake in which we assess whether an AI POC is a good fit, which type would work best and which scope is realistic within your timeline.

Edit content