Streaming STT Latency on Serverless GPU: A Budget

Streaming speech-to-text (STT) latency over WebSockets on serverless GPU behaves like a budget, not a reading on a single gauge. The number a user feels is the sum of four terms, and each one responds to a different lever. Optimise the one that is already small and you waste effort; miss the one that dominates and your voice agent stutters. This guide treats end-to-end latency as an explicit budget a Lead ML Engineer can allocate, measure, and defend.

Why streaming STT latency is a budget, not a number

The industry target for streaming transcription is well established. Deepgram frames the goal as sub-300ms end-to-end latency, while AssemblyAI defines real-time STT as a live audio stream converted to text over a persistent WebSocket with latency under one second. Those are envelopes, not line items. To hit them reliably you have to decompose the number into components you can each attack: transport, inference, emission, and the cold-start tax. For a broader view of where milliseconds leak across a pipeline, see our write-up on network latency in the AI inference pipeline.

The four-component latency budget

1. Transport - the WebSocket round trip for audio up and transcripts down.

2. Inference / RTF - how fast the model turns audio into tokens, governed by real-time factor.

3. Emission - when partial versus final transcripts are released, which splits perceived from committed latency.

4. Cold-start tax - the one-time penalty when a serverless GPU spins from zero, which can dwarf every other term combined.

Budget each term separately, then add them. The sections below give the short, complete answer for each.

WebSocket transport

WebSocket is the default transport for streaming STT because audio and recognition results must travel concurrently over one persistent, bidirectional connection. ElevenLabs describes it as "a long-lived, ordered, reliable, bidirectional channel over TCP" that traverses the proxies and firewalls already allowing HTTPS. A good setup delivers audio "in the low hundreds of milliseconds, from spoken word to your app," with MeetStream citing roughly 200ms for live WebSocket streaming.

The practical mechanics: send audio in small binary chunks at real-time rate. Deepgram slices audio into 100-200ms chunks so the recognizer never waits for a full sentence; Soniox notes the tradeoff plainly - "smaller frames mean lower latency and more messages. Larger frames mean fewer messages and slightly more lag." Do not dump a whole file down the socket at once; as Soniox puts it, that "is an upload wearing a costume," and late chunks leave the recognizer idle.

One caveat on reliability: WebSocket runs over TCP, so "every byte arrives in order, even if that means waiting" - head-of-line blocking under packet loss, where WebRTC over UDP would conceal the drop. For most server-to-server STT this is an acceptable trade.

Partial and final transcripts

Emission timing splits perceived latency from committed latency. A partial result is "the recognizer's current best guess" and can still change; a final result "is committed and will not change." Render partials fast for responsiveness, but, per Soniox's rule that prevents most live-transcription bugs, "render partials, but act on finals." Time-to-first-token - the elapsed time from speaker audio to first transcribed words - is the perceived-latency metric worth instrumenting; leading providers report it in the 64-150ms range, with Smallest AI's Pulse at 64ms and ElevenLabs Scribe v2 Realtime around 150ms over WebSockets.

Real-time factor and GPU inference

Inference latency is governed by real-time factor (RTF): processing time divided by audio duration. As Dialpad explains, five seconds of audio transcribed in five seconds is an RTF of one - "for these systems, we are generally looking for RTF values that are less than one." An RTF at or above 1.0 means the backlog grows and latency climbs without bound, so RTF < 1.0 is the non-negotiable condition for true streaming. There is a quality dimension too: arXiv work on fragmentation shows VAD-based splitting gives the best quality at the highest delay, while fixed-interval chunking gives the lowest delay at lower quality. Running the model on a GPU you control is what buys headroom here. On Whisper specifically, our guide to maximising Faster-Whisper performance covers the inference tuning, and our Ultravox deployment shows a full speech pipeline reaching 600ms first-time-to-audio end to end.

The serverless GPU cold-start problem

This is the term that sinks voice agents, and it is the one angle most latency guides ignore. A serverless GPU cold start is "a stack of sequential delays, container pull, model-weight load, and CUDA graph warmup, that can take 30 to 90 seconds." Container image pull dominates - 4-8 minutes uncached for 15-18 GB images - while weight loading runs ~8s for small models and 40-45s for 70B, and CUDA graph warmup adds 10-30s. The gap is stark: cold starts exceeding 40 seconds to first token against ~30ms per token warm, a 1000x latency gap. Against a voice turn's sub-second budget, "a multi-second gap is a dropped call."

The mitigation is to never serve a live turn from a cold container: keep warm pools / reserved replicas so a GPU is always hot, and shrink the phases that remain. We cover the full anatomy and fixes in serverless GPU cold starts for voice AI, and the weight-load phase specifically in loading model weights from object storage.

Setting production latency targets

Self-hosting the STT model next to the rest of the pipeline is what collapses the transport-plus-inference terms. In Cerebrium's own testing we measured Deepgram STT at ~110ms TTFB hosted on Cerebrium versus ~250ms through the hosted API - a 140ms saving on a single term. A reasonable production allocation, built only from figures above:

Putting it together: reference flow

Audio is captured and framed into ~100ms binary chunks, pushed up a persistent WebSocket at real-time rate. A warm serverless GPU runs the STT model at RTF < 1.0, emitting partials for the UI and finals for downstream actions. Because the GPU is held warm, the cold-start tax is zero on every live turn. Size the fleet from concurrency per GPU - our concurrent voice sessions per GPU numbers are the input - and if STT is one stage of a full agent, our Pipecat voice agent GPU deployment guide stitches the stages together, while speech-to-speech vs cascaded latency covers the architectural choice above it. For a full 500ms-latency reference agent, see deploying a global-scale voice agent.