When you wire up a voice feature using a cloud speech-to-text API, something happens that rarely gets discussed in sprint planning: every word your users speak has to leave the environment where it was captured and travel to someone else's infrastructure for processing.
What happens to that audio from there — whether it is retained, for how long, and for what purposes — depends on the provider, product configuration, and contract.
For a to-do app or a fitness tracker, that might feel like an acceptable tradeoff. For a clinical notes tool, a legal dictation product, or frankly any app with users who've read the news lately about AI data practices, it's a much more consequential decision.
Offline transcription has become genuinely practical, and a large part of that ecosystem rests on OpenAI's Whisper, an openly released speech-recognition model that can be run without sending recordings to an external transcription API.
It's worth pausing on what that actually means for product teams building voice features right now.
The default path is understandable. You grab an API key, send an audio blob, get back a transcript. It's fast to ship, the accuracy can be good, and you don't have to think about model hosting.
I've watched plenty of teams take this route because the alternative felt like a research project rather than a product decision.
But the tradeoff isn't just philosophical. It's structural.
When a cloud STT provider processes personal data on your behalf, that can introduce another processor into your data flow, along with the contractual, security, and accountability obligations that follow. Under GDPR, the precise controller/processor relationship depends on what each party actually does with the data, but where a provider is acting as your processor, the Article 28 framework becomes relevant.
International transfers can introduce another layer depending on where processing occurs and what transfer mechanism applies.
The ICO's guidance on controllers and processors covers these responsibilities, while the EDPB's Guidelines 07/2020 go into considerably more detail on how controller and processor roles are determined.
None of that makes cloud transcription inherently non-compliant. Plenty of organisations use cloud processors lawfully.
But every additional party and transfer becomes another relationship your product team has to understand, document, secure, and govern.
For US healthcare products subject to HIPAA, the implications are different but equally concrete. If a covered entity or business associate sends electronic protected health information to a cloud service for processing, HHS generally treats that cloud provider as a business associate. That means an appropriate Business Associate Agreement is required.
The HHS Office for Civil Rights guidance on cloud computing and HIPAA is useful reading here.
None of these problems are unsolvable through contracts, security controls, and compliance processes.
But they add drag, create ongoing obligations, and, crucially, don't change the fundamental architecture: the recording still has to leave the environment where it was captured in order to be transcribed.
OpenAI released Whisper along with model weights and inference code in 2022, and the ecosystem around running it locally has matured considerably since then.
There are actually two privacy architectures worth distinguishing here.
One is local or self-hosted transcription, where your organisation runs the model rather than sending audio to an external STT provider.
The other is true on-device transcription, where inference happens on the phone, tablet, or computer that captured the audio and the recording never has to leave it.
Two projects are particularly relevant:
whisper.cpp is a C/C++ implementation of Whisper designed to run efficiently across a wide range of hardware. It supports CPU inference, Apple Silicon acceleration through technologies including Metal and Core ML, NVIDIA GPUs through CUDA, and platforms including macOS, Windows, Linux, iOS, and Android. This makes it particularly interesting when you want to embed transcription directly into an application and keep inference on the user's device.
faster-whisper reimplements Whisper using CTranslate2. It is designed for substantially more efficient inference than the original Python implementation and is particularly useful when you want to run Whisper locally or on infrastructure you control.
That distinction matters.
faster-whisper running on your own server can remove an external speech-to-text provider from the architecture, but the audio still leaves the user's device.
whisper.cpp can support the stronger architecture: the recording can be transcribed on the device that captured it without being sent elsewhere at all.
"Your recordings never leave your device" is a materially different proposition from "our transcription provider is contractually prohibited from retaining your recordings."
The second is a policy and contractual assurance. The first can be an architectural property of the product.
With true on-device inference, the transcription step itself does not require sending the recording to an external STT provider. That can remove an entire third party from the audio-processing path — and with it, the processor relationship, transfer considerations, and retention questions that particular provider would otherwise introduce.
It does not make the application automatically GDPR- or HIPAA-compliant.
What you do with the resulting transcript still matters. If your app uploads it, synchronises it between devices, backs it up to the cloud, sends it to another AI model, or stores it in your own backend, those data flows still have to be designed and governed appropriately.
But you've eliminated one data movement that didn't necessarily have to exist.
That's a meaningful difference.
The Whisper family also gives developers a useful range of model sizes. OpenAI's released models range from the 39-million-parameter Tiny model through Base, Small, and Medium to the 1.55-billion-parameter Large family, with the later Turbo model offering another performance/speed tradeoff.
Smaller models require fewer resources but generally trade some recognition quality for speed. Larger models demand considerably more compute and memory.
Quantisation can reduce model storage and memory requirements substantially, making smaller Whisper variants much more practical on constrained hardware, although the exact savings depend on the implementation and the precision you're starting from.
There is no universally correct model size. The right one depends on what hardware you need to support, what languages and accents your users speak, what kind of audio they're recording, and how much latency they will tolerate.
Imagine a dictation app built for GPs or solicitors who need to capture notes hands-free between appointments.
Both sectors routinely deal with highly sensitive information. Sending every recording to an external transcription provider creates another data flow the product team has to actively manage, secure, document, and explain to clients.
On-device transcription can remove that particular problem at the source.
In my experience, this is exactly the kind of architectural decision that changes how a client can pitch their own product.
"Your recordings never leave your phone."
That's a very different statement from:
"Your recordings are sent to our transcription provider, but our contract says they won't retain them."
For a GP practice evaluating a product, for example, the first architecture may be materially easier to explain internally than the second. It doesn't remove the need for proper information governance, but it reduces the number of systems through which the original recording has to travel.
The engineering tradeoffs are real and worth stating plainly.
On lower-powered devices, the smallest Whisper models may be practical while Medium or Large variants may feel unacceptably slow. On Apple hardware, implementations such as whisper.cpp can take advantage of Apple Silicon acceleration through Metal and Core ML, making local inference considerably more practical.
This is why I'd encourage any team considering this path to benchmark their actual target hardware early, with actual audio samples from their domain, rather than relying on general benchmarks that may bear little resemblance to medical dictation, legal terminology, regional accents, noisy rooms, or the microphones their users actually have.
Latency figures vary too much across hardware and model configurations to quote responsibly as a universal rule.
What matters is whether your specific target device produces acceptable results for your specific audio conditions.
That's a prototyping exercise, not a research programme.
Cloud STT APIs are commonly priced according to audio usage. At low volumes, the cost can be negligible.
At the volumes a genuine business generates, the bill grows, and it grows alongside user engagement, which is a peculiar incentive structure for a product you want people to love.
On-device inference changes the economics.
Instead of paying an external provider for every minute of transcription, you invest in integration, model distribution, testing, and maintenance while much of the inference compute happens on hardware the user already owns.
That does not make on-device inference free.
You still have engineering costs. Model files have to be distributed and updated. Device fragmentation has to be tested. Local inference consumes storage, memory, CPU or GPU resources, and battery power. Supporting several hardware classes can become its own maintenance burden.
But at sufficient transcription volume, removing a recurring per-minute API charge can become economically significant.
There isn't a universal crossover point.
The honest answer is to run the numbers for your own product: projected transcription volume, provider pricing, target devices, engineering effort, support costs, and whatever infrastructure remains in your architecture.
The cloud provider's pricing page is one input.
It isn't the whole calculation.
On-device transcription isn't a complete answer to every voice-feature problem. A few things it won't help with:
Accuracy and hallucination: Local processing solves a data-flow problem, not an accuracy problem. Whisper can still mis-transcribe speech and, in some circumstances, generate text that was not actually present in the audio. OpenAI itself documents this limitation. That matters enormously in medical and legal workflows. High-consequence transcription should include appropriate user review and validation against the actual environments in which the product will be used.
Model updates: If a substantially better model becomes available, getting that update to your users requires an application update or an on-device model download. Neither is as frictionless as replacing a model behind a server-side API.
Very low-end devices: Whisper's smaller models can be surprisingly practical, but there's a hardware floor below which the experience degrades. If your target audience includes genuinely budget-tier devices, test thoroughly rather than assuming "offline" automatically means "works everywhere."
Storage and distribution: Hundreds of megabytes or more of model data may be perfectly acceptable on a desktop workstation and considerably less welcome inside a mobile application. Model downloading, versioning, caching, and cleanup become product concerns.
Battery and thermals: Moving inference onto the user's hardware also moves the compute workload there. Sustained transcription can affect battery consumption and device temperature, particularly on mobile hardware.
Custom vocabulary and domain adaptation: Medical terminology, legal language, product names, and specialist vocabulary can expose weaknesses that aren't obvious on generic speech benchmarks. Cloud platforms may offer domain-specific features that require additional engineering to reproduce locally.
The transcript still has to be protected: Keeping the original audio on-device does not make the resulting text harmless. A clinical or legal transcript can be just as sensitive as the recording that produced it. If the transcript is synchronised, backed up, analysed, or sent to another AI system, those flows still matter.
These are real constraints.
They don't invalidate the approach; they just mean this is an architectural decision rather than an obvious default swap.
I have come to believe, after a fair amount of time building software products, that privacy-by-architecture is a fundamentally different thing from privacy-by-policy.
A policy tells users what you intend to do with their data. Architecture determines what's even physically possible.
When a recording is processed entirely on-device, you have removed one whole category of exposure: the audio does not need to exist on a transcription provider's infrastructure at all.
There is no provider-side copy that needs a retention policy. No additional transcription service through which the recording has to pass. No external STT provider whose handling of that audio becomes another part of your threat model.
That doesn't eliminate the responsibility to secure the recording and transcript wherever your application stores them.
It simply reduces the number of places they have to exist.
And I think that distinction is going to matter increasingly to the people signing up for and paying for software products.
As AI capabilities become easier to integrate, the architectural question shouldn't automatically be:
"Which AI API should we use?"
Sometimes the better question is:
"Does this data need to leave the device at all?"
The maturation of open-source offline transcription changes the burden of proof. For products handling particularly sensitive speech, sending every recording to an external API should no longer be an unquestioned default.
It should be an architectural choice that the team can explain and justify.
That's a healthier way to think about privacy: not as a promise written after the system has been built, but as one of the constraints that determines how the system is built in the first place.
The regulatory references in this article describe general frameworks and are not legal advice. Data protection and health-data compliance obligations vary by product, jurisdiction, architecture, and user base. Take qualified legal advice before shipping any product that handles sensitive personal data.
Offline speech recognition makes it increasingly practical to build voice features without automatically sending every recording to an external transcription provider. Self-hosted Whisper can reduce dependence on third-party STT services, while true on-device inference can keep the original recording on the device that captured it. That doesn't automatically solve GDPR, HIPAA, security, or accuracy concerns, but it can materially reduce the number of systems through which sensitive audio has to travel.
Yes, if inference genuinely runs locally and your application does not separately upload the recording. An implementation such as whisper.cpp can perform transcription on the user's hardware without sending the audio to an external STT API. Be careful to distinguish this from self-hosting: running Whisper on your own server removes the third-party transcription provider, but the audio still leaves the user's device.
It can be, but there is no universal answer. Accuracy depends on model size, language, accent, microphone quality, background noise, domain vocabulary, and hardware constraints. Whisper can also produce transcription errors and occasional hallucinations. The right approach is to benchmark the models on your actual target hardware using representative audio before committing to an architecture.
Local inference can simplify the data flow by removing an external transcription provider from that particular processing step. That may remove processor, transfer, retention, or BAA considerations associated with that provider. It does not make the overall application automatically compliant. How recordings and transcripts are stored, synchronised, analysed, shared, and secured still matters.
On-device transcription happens on the phone, tablet, or computer that captured the audio, so the recording does not need to leave that device for transcription. Self-hosted transcription runs on infrastructure controlled by your organisation. It can avoid sending data to an external STT provider, but the audio still travels from the user's device to your infrastructure.
There is no universal threshold. On-device inference removes recurring per-minute API charges but introduces engineering, testing, model-distribution, storage, device-support, and maintenance costs. The right comparison uses your expected transcription volume, target hardware, provider pricing, and development costs.
Projects such as whisper.cpp and faster-whisper have made local Whisper inference far more accessible than building a speech-recognition system from scratch. A competent engineering team can prototype local transcription without doing ML research, although production deployment — particularly across a wide range of mobile devices — still requires careful performance, accuracy, security, and UX testing.
It can materially reduce the data surface because the original audio does not need to pass through an external transcription service. But privacy depends on the entire application architecture. If the transcript is subsequently uploaded to your backend, synchronised to the cloud, sent to another AI service, or included in analytics, those data flows still need to be considered.
