Custom Whisper transcription integration development

Q: Can Whisper handle real-time streaming?

Not out of the box, as Whisper is originally a batch model. For real-time use, we build a streaming layer with Voice Activity Detection (VAD) that splits audio into chunks of 5-10 seconds, sends these in parallel to faster-whisper and merges the outputs with overlap correction. End-to-end latency then comes to 1-3 seconds, sufficient for live captioning and conversational AI. For sub-second latency, streaming-native models (Deepgram Nova, NVIDIA Parakeet) are usually more suitable.

Q: How does diarisation (who speaks when) work?

Whisper gives you what was said, but not speaker labels. For that we combine it with pyannote.audio: it extracts speaker embeddings from the audio and clusters them into speaker segments. The Whisper transcript and the pyannote segments are merged on timecode, resulting in a transcript with 'Speaker 1', 'Speaker 2' and so on. With a few seconds of reference audio per person, we can also replace the labels with real names.

Whisper is OpenAI's open-source speech recognition model. It understands over 99 languages and can run locally or in the cloud. We build Whisper integrations for businesses that want to transcribe, search or analyse audio and video streams with AI, from meeting recordings and client calls to podcasts, video platforms and legal dictation. No vendor lock-in, full control over your data, and integration with your existing systems.

Whisper Large-v3 faster-whisper Real-time streaming Diarization Multilingual ASR

Discuss your transcription case View applications

What Whisper is and why it is becoming the standard for ASR

OpenAI released Whisper in 2022 as an open-source Automatic Speech Recognition model, trained on 680,000 hours of multilingual audio. Since then it has become a reference architecture for speech-to-text, with Large-v3 as the current top model. The combination of openness, broad language coverage and competitive Word Error Rate (WER) makes Whisper attractive to organisations that do not want to be tied to a closed cloud API.

Unlike Azure Speech, Google Cloud Speech-to-Text, Deepgram or AssemblyAI, Whisper is open source under the MIT licence. You can host the model on your own infrastructure, fine-tune it on domain-specific audio and use it without per-minute pricing. The trade-off is that you take responsibility for inference speed, GPU capacity and streaming architecture yourself. For many businesses that is precisely the preferred trade-off, since control over data and costs outweighs the convenience of a managed service.

Whisper comes in several sizes: tiny, base, small, medium, large-v2 and large-v3. The larger the model, the lower the WER and the better the robustness to background noise, accents and code-switching (mixing languages within a single conversation). For production we usually choose Large-v3 on GPU, or the optimised variants faster-whisper and whisper.cpp when CPU-only operation is a requirement. Which variant suits you depends on latency requirements, audio quality and budget.

The official Whisper source code is available at github.com/openai/whisper. We use it as a foundation but build a production layer around it: REST/WebSocket APIs, queue management, retry logic, metadata storage, Voice Activity Detection and post-processing. This means the model doesn't run as an isolated script but as an integral part of your application.

Applications for which Whisper integrations are built

From internal meeting tooling to public media platforms, Whisper provides the transcription layer on which further AI functionality (summarising, searching, analysing) can be built.

📋

Meeting transcription and minutes AI

Meeting recordings from Microsoft Teams, Zoom, Google Meet or your own telephony system are automatically transcribed, diarised (who speaks when) and summarised by an LLM into action points, decisions and open questions. The note-taker shifts from typing verbatim to reviewing and polishing.

📞

Call centre QA and sentiment

Customer conversations in a contact centre are transcribed and analysed for sentiment, compliance statements (for example, correctly delivering a disclosure obligation) and first-call resolution. QA teams no longer sample 2% of calls; they get 100% in a searchable dashboard.

🎙️

Podcast and video platforms

A media platform lets editors or users upload audio or video and automatically delivers transcripts, chapter markers and SEO-friendly show notes. Whisper transcribes in 99+ languages, so international content flows through the pipeline without extra workflows.

⚖️

Legal and medical dictation

Lawyers dictate memos instead of typing them, and doctors record patient histories by voice straight into the EHR. Fine-tuning on specialist terminology (case law, ICD-10 codes, medication names) brings the word error rate (WER) well below that of generic ASR. On-premise hosting is often mandatory here because of professional confidentiality.

🎬

Auto-captions and accessibility

News broadcasts, e-learning platforms and webinars produce subtitles automatically. Whisper delivers timestamped segments that can be exported directly as WebVTT or SRT. For live streams (RTMP/HLS), we build a streaming pipeline with chunking for near-real-time captions that meet WCAG guidelines.

💼

Interview summarisation for research

UX researchers, journalists and consultants work through stacks of interviews. Whisper transcribes, an LLM produces thematic summaries and pulls quotes by topic. The researcher searches by concept ("complaints about onboarding") and immediately gets the verbatim excerpts with timecode and speaker.

How Appfront builds a Whisper integration

A Whisper integration is more than a wrapper around model.transcribe(). In production, this adds audio ingestion (from a telephony system, video CMS, browser recorder or WebRTC stream), voice activity detection to trim silences, diarisation to separate speakers, a batch or streaming architecture, error handling, and integration with your existing systems: CRM, EHR, DMS, video platform or analytics dashboard.

We start with the audio reality of your use case. Short structured dictations? A batch API using faster-whisper on a single GPU will suffice. Two-hour meetings? We split them into overlapping chunks, parallelise across multiple workers and restore the timecodes in a post-processing step. Live captioning for an event? We build a WebSocket streaming pipeline with VAD and chunked inference that returns text within one to three seconds.

For diarisation we integrate pyannote.audio — an open-source toolkit that extracts and clusters speaker embeddings. Combining Whisper for what is said with pyannote for who says it gives you a transcript that is immediately usable for minutes, legal records or analysis. For languages with code-switching (Dutch with English technical jargon, or Moroccan Arabic with Dutch), we configure Whisper in multilingual mode and add language detection per segment.

Building a speech recognition app AI document processing Building an AI summarisation tool AI sentiment analysis Smart API integrations

From first audio sample to production pipeline

Our approach to Whisper integrations follows four phases. Each phase ends with something concrete: not abstract reports, but working components you can test.

Audio assessment

We listen to representative samples from your use case: quality, speakers, background noise, languages and jargon. Based on this, we choose the model size, hosting arrangement and any fine-tuning. Output: a well-founded architecture decision.

Proof of concept

Within a few weeks, a working prototype is running that processes your audio and produces a structured transcript. We measure WER on your own test set, so you can objectively verify that the quality meets your standards before you proceed.

Integration and hardening

We make the pipeline production-ready: queue management, retries, monitoring, logging, security and integrations with your CRM, EHR, video platform or dashboard. Optionally with streaming via WebSocket for live use cases.

Ongoing management

New Whisper releases, model updates, scaling moments and vocabulary extensions all require maintenance. We monitor WER, latency and cost per minute of audio, and adjust when it is worthwhile.

Implementation choices and the stack we deploy

The main choice is whether you run Whisper as a managed service (OpenAI Audio API), self-hosted on CPU or self-hosted on GPU. With OpenAI Audio you pay per minute of audio and send files to US infrastructure: easy to set up, but not always compatible with GDPR requirements or predictable costs at high volumes. With whisper.cpp, Whisper runs on pure CPU thanks to a GGML implementation: ideal for batch work on existing servers, edge applications or on-premises deployments where a GPU is not available.

For production volumes with latency requirements, we almost always choose faster-whisper — a reimplementation based on CTranslate2 that runs Whisper Large-v3 four to eight times faster on a single consumer graphics card than the reference implementation. For very high throughput, we deploy Whisper on vLLM or Triton with batching, which drastically lowers our inference cost per minute of audio at volumes above several thousand hours per month.

For live captioning or conversational AI we build a streaming pipeline: audio arrives via WebRTC or WebSocket, Voice Activity Detection (VAD) splits it into speaker segments, faster-whisper transcribes chunks of 5-10 seconds with overlap, and the text streams back to the client. For post-processing (summaries, action items, entity extraction, translation) we connect the transcript to an LLM such as GPT-4, Claude or a locally hosted Llama model.

Whisper Large-v3 faster-whisper whisper.cpp CTranslate2 pyannote.audio VAD (Silero) WebSocket streaming WebRTC FFmpeg FastAPI Python PyTorch vLLM Docker Kubernetes Redis Queue

Speech, storage and privacy: what applies to your transcriptions

Audio recordings are personal data and, in many cases, special category personal data. Whisper integrations are subject to stricter rules than typical data pipelines. We design alongside you within the legal and sector-specific frameworks.

GDPR: consent, purpose limitation and retention periods

Voice recordings can be traced back to individuals and fall under the GDPR. We help you set up consent flows (prior notice, opt-in where required), explicit purpose limitation and automatic deletion policies. Audio and transcripts often have different retention periods, and our pipelines support that.

NEN 7510 for healthcare providers

For hospitals, mental health institutions and GP out-of-hours services, NEN 7510, NEN 7512 and NEN 7513 apply. We host Whisper pipelines for patient histories, dictation or patient communication locally or in a NEN-certified environment, with audit logging on every transcription action.

EU hosting and data residency

For clients who cannot or do not wish to use the OpenAI Audio API, we host Whisper on Dutch or European cloud providers (Hetzner, Scaleway, OVH, Leaseweb, Microsoft Azure West Europe). Audio and transcripts never leave the EU. We document the data flows in a register of processors.

Professional secrecy and confidentiality

Lawyers, doctors and notaries are bound by statutory professional secrecy. For them, we work with on-premises inference, end-to-end encryption and strict access control. Audio is encrypted at rest and in transit, with key management kept under the client's control.

Privacy-first analytics Scrive eIDAS e-signature AI in the healthcare sector

Real-world scenarios

Whisper integrations only deliver value once they fit into your workflow. Below are four realistic examples that cover most enquiries.

AI meeting minutes for a professional services firm

Advisers conduct client calls on Teams every day. A server-side bot joins each meeting, records audio (with consent), runs Whisper Large-v3 with pyannote diarisation, and sends the transcript to an LLM that extracts action items, decisions and open questions. The result appears automatically in the CRM, linked to the account, ready for the adviser to review.

Compliance monitoring in a contact centre

An insurer has 200 call centre agents and wants to transcribe 100% of conversations to monitor compliance (duty to inform) and sentiment. Faster-whisper runs on a GPU cluster, call recordings from Genesys are processed in batches, and a dashboard shows per agent and per topic how often mandatory statements were made correctly.

Live captions for an events platform

A hybrid event organiser wants speakers in Room 1 (Dutch) to be captioned live for deaf and hard-of-hearing attendees in the room and for remote visitors. A WebSocket pipeline with VAD and faster-whisper delivers text within 1.5 seconds, displayed at the bottom of the stream. Translation into English runs in parallel via an MT layer.

Searchable video archive for a knowledge institute

A research centre holds 8,000 hours of historical video recordings from congresses and interviews. We build a batch pipeline that transcribes and diarises everything and indexes the text in OpenSearch. Researchers search by keyword and jump straight to the right timecode in the video, turning "somewhere on a tape" into "found within five seconds".

Why choose Appfront for your Whisper integration

End-to-end audio pipeline

We build not just the transcription component but the entire chain: audio ingestion, VAD, diarisation, Whisper inference, LLM post-processing, storage and integration with your CRM, EHR or platform. One partner for the whole AI layer on top of your audio.

Open-source and managed expertise

We know the whole Whisper ecosystem: from whisper.cpp on edge devices to faster-whisper on GPU clusters to vLLM/Triton for serverless scale. We choose based on your use case, not on what we happen to already have in place.

Privacy-first design

For healthcare, legal, government and finance, we build as standard with EU hosting, on-premises as an option, audit logging and strict data retention. GDPR, NEN 7510 and sector-specific frameworks are built in from the start, not bolted on afterwards.

Get in touch Building an MVP Building a web app

Frequently asked questions about Whisper transcription integration

When should I choose Whisper, and when should I choose Azure Speech, Google Speech, Deepgram or AssemblyAI?

Whisper is open source under the MIT licence, multilingual (99+ languages) and self-hostable. That is decisive when you want data sovereignty, predictable costs at high volumes or fine-tuning on your own jargon. Managed services (Azure, Google, Deepgram, AssemblyAI) win on time to first result and out-of-the-box streaming functionality, but they make you dependent on a vendor and send audio to US infrastructure. For medical or legal data, or for government, Whisper is usually the better choice; for a simple podcast MVP without volume requirements, a managed service may be quicker.

Does Whisper work well in Dutch?

Yes. Dutch is a top-tier language for Whisper in terms of training volume. On typical Dutch audio (interviews, podcasts, meetings), Whisper Large-v3 achieves a Word Error Rate that is competitive with commercial Dutch ASR engines. For domain-specific jargon (medical, legal, technical), fine-tuning on a few hundred hours of your own audio helps.

Can Whisper handle real-time streaming?

Not out of the box: Whisper is originally a batch model. For real-time use, we build a streaming layer with Voice Activity Detection (VAD) that splits audio into chunks of 5-10 seconds, sends these in parallel to faster-whisper and merges the outputs with overlap correction. This brings end-to-end latency to 1-3 seconds, sufficient for live captioning and conversational AI. For sub-second latency, streaming-native models (Deepgram Nova, NVIDIA Parakeet) are usually a better fit.

How does diarisation (who speaks when) work?

Whisper tells you what was said, but not who said it. To add that, we combine it with pyannote.audio: this extracts speaker embeddings from the audio and clusters them into speaker segments. The Whisper transcript and the pyannote segments are merged on timecode, resulting in a transcript with "Speaker 1", "Speaker 2" and so on. With a few seconds of reference audio per person, we can also replace the labels with real names.

How do you handle multilingual recordings and code-switching?

Whisper detects the language automatically from the first segment. For recordings where languages are switched within a single conversation (Dutch with English technical jargon, or Moroccan Arabic with Dutch), we configure Whisper in multilingual mode and run language detection per chunk rather than once. The output then includes a language tag per segment, which enables downstream processing (translation, summarising into one target language).

May I use Whisper for patient recordings, and what about NEN 7510?

Yes, provided you run Whisper on-premises or in a NEN 7510-certified environment. In most cases, the OpenAI Audio API is unsuitable for patient data because of US data transfer and the lack of NEN certification. We would then host Whisper Large-v3 within your own infrastructure or with a Dutch hosting provider that supports NEN 7510, with audit logging of transcription actions and strict access control in line with NEN 7512.

How accurate is Whisper, and how do you measure that?

Accuracy is expressed as Word Error Rate (WER): the percentage of words that are wrong, missing or added compared with a manual reference transcript. We measure WER on a dedicated test set of your audio (typically 30-60 minutes across different speakers and conditions), so you have an objective figure. Whisper Large-v3 on clear Dutch audio achieves a WER below 10%; with background noise, overlapping speakers or jargon, this can rise.

How long does it take to build a Whisper integration?

A proof of concept on your own audio typically takes a few weeks. A production pipeline with integration into your CRM, EHR or video platform, plus monitoring and compliance, takes longer, depending on scope. We work in iterative sprints so you see interim results and can steer as we go, with no big-bang delivery at the end.

Turning audio and video into text, insight and action?

Discuss your transcription case with us. We'll listen to your audio, choose the right Whisper architecture and build a working prototype. No-obligation and free of commitment.

Schedule a conversation