In real-time voice interaction, latency is the only deadline. A text answer a few hundred milliseconds late leaves the user mildly unimpressed. One audible stutter and the sense of talking to something non-human destroys trust immediately. To fix that engineering problem, OpenAI spent six months rebuilding its entire voice system.
OpenAI has now published an engineering write-up on GPT-Live setting out the architectural changes behind the new ChatGPT voice stack. The single most notable figure: the new media system’s p95 audio-frame latency has fallen to the level of the old system’s p50. The slowest 95 per cent of frames in the new system now run as smoothly as the fastest half in the old one. No absolute millisecond figures were given, but the result shows the work targeted the occasional, conspicuous slow frame rather than the average.
GPT-Live no longer waits for a user to finish a sentence before starting work. Audio flows into the model continuously and generated speech flows back continuously. Search, tool calls and complex reasoning are moved to a separate asynchronous path.
Separating the traffic
The first thing changed was how audio moves inside the server. For years the standard approach treated a voice agent as a single pipeline of speech to text, model inference, then text to speech, with audio processing, model calls, tool requests and chat-history writes all running in the same asynchronous service. Slow down any one stage and everything behind it queues.
That is tolerable in text products. Audio frames cannot queue. Every frame has a playback position, and once it is late, finishing the work may be pointless. Once old audio starts to pile up, later audio gets slower too and the whole conversation drifts behind where the user actually is.
OpenAI built a multi-path real-time transport network modelled on the layered latency channels of the human nervous system. A fast channel handles reflexive responses: barge-in detection and back-channel sounds such as acknowledgements. It is extremely latency-sensitive and is usually served by tiny models or hard-coded logic at the edge or on the client. A deep channel decouples real-time exchange from complex reasoning, with models such as GPT-5.5 handling semantic understanding and long-range inference. Asynchronous work, including search, tool calls and data persistence, is removed from the main path completely. Background tasks may return late. They may not stall audio.
The media front end and part of the inference logic were rewritten from Python asyncio into Go. This is not a simple argument about which language is faster. Real-time audio handles enormous numbers of very small, extremely time-sensitive UDP packets, and Python’s thread scheduling, memory allocation, data copying and unpredictable garbage-collection pauses under high concurrency are precisely what manufactures p95 latency.
The work went down into the Linux kernel as well. SO_REUSEPORT lets multiple workers share one UDP port with the kernel balancing load. Goroutines reading UDP are pinned to operating-system threads to reduce thread migration and CPU cache invalidation. Receive buffers are pre-allocated to cut memory copying. The point of all of it shows up in the tail: the new system does not merely lower average latency, it makes the overwhelming majority of audio frames arrive on time.
Six round trips down to one
At the network layer, establishing a standard WebRTC connection takes six round trips. For cross-region connections, the speed of light alone is enough to produce a noticeable delay before the first word.
OpenAI built a custom protocol called WARP that merges the DTLS handshake, SCTP setup and data-channel negotiation. Connection start-up drops from six round trips to one.
Routing hints are written into the ICE ufrag that WebRTC already carries, so the routing information travels inside the connection protocol itself. When the relay layer receives the first packet it knows which instance to forward to without querying a remote Redis, and can build the mapping in memory, removing a cross-network lookup entirely.
Cutting six round trips to one does not make start-up six times faster overall, since server scheduling, packet loss, client processing and model warm-up still take time. It does remove several mandatory waits on the network. For cross-region connections, eliminating full round trips is generally worth more than shaving further milliseconds off server-side code.
Who is speaking, and what did the user actually hear
With transport stabilised, the harder problem is how the model manages turn-taking and session state during a continuous conversation. Older voice systems used a separate turn detector, judging from silence duration whether a user had finished before waking the main model. GPT-Live moves that judgement inside the speech model. Audio flows in continuously and the model decides, as it understands, whether to keep listening, start answering, pause output or accept an interruption. That allows pauses to be read through semantics, tone and context, but it means the main model runs continuously through the session. OpenAI has not disclosed how much compute that adds, nor published data on false starts and wrong interruptions.
Interruption is the hardest part. A user may cut in at the four-second mark while the model has already generated out to ten seconds, with some audio already sent to the client. The system cannot simply stop generating. It has to track separately where the model generated to, where the server transmitted to, and what the user actually heard. The next turn must be built only from what was genuinely heard, or the model will assume it has already said things the user never received. OpenAI has not published its playback-confirmation or audio-rollback protocols, but notes that the text, time range and speaker attribution of the most recent message all remain editable. Model output is not final record until interruption and actual playback are reconciled.
Continuous voice also demands that model instances migrate without breaking the session. A long conversation holds context and KV cache. Switching to a blank instance would force reprocessing of the entire history and produce an audible gap. OpenAI keeps the old instance running while a new one starts and completes prefill. The new instance then catches up on audio that arrived during preparation, and only once it has caught up does the system switch the media stream. Context compression uses the same mechanism: the old instance keeps talking while history is compressed in the background. It avoids visible interruption at the cost of briefly paying for two sets of inference. How much of long-term instruction, unfinished tasks and tool state survives repeated compression has not been disclosed.
Results that arrive too late to be right
The genuine difficulty in a dual-model architecture is not handing work to the background. It is making sure the answer still fits the conversation when it comes back. Once GPT-5.5 starts a search or a tool call, GPT-Live keeps listening and responding. In the meantime the user may add conditions, change the question, or cancel the original task. A correct answer from the background model may no longer apply.
Every background task therefore has to be bound to the conversational position where it started. When a result returns, the system checks whether the current dialogue still carries the original intent before playing anything. It works like a resumable transfer with state validation: the background model starts from a session checkpoint and, on completion, confirms whether its output can safely be spliced back into a conversation that has moved on. OpenAI has not published its handling of task versions, cancellation signals or stale results, but those mechanisms determine whether the system reads out answers that have already expired.
A second mismatch runs through the product. The model processes continuous sound, while ChatGPT’s search, logging, safety and chat history need discrete messages. User and assistant may speak at once, a short acknowledgement may not deserve its own message, and a few interjected words may not be a real interruption. The application server maintains a mutable provisional record first, then confirms final messages using timing, transcription and turn ownership. The interface uses the faster speculative state so captions appear promptly, while logging, analytics and parts of the safety system rely on the more stable authoritative record so background tasks can find reliable checkpoints.
Before launch OpenAI ran shadow testing, feeding real voice sessions into both old and new systems at once. The tests found an auxiliary component saturating earlier than expected and dragging down the inference queue, a reminder that stable handover between two models depends on the network, the queues and the state services keeping up, not only on model speed.
Read as a whole, the point of GPT-Live is not that one model got faster. OpenAI redivided responsibility inside a real-time voice system. Audio was placed on its own fast path, the WebRTC entry point was split into relay and transceiver, routing information was pushed into the connection protocol, model instances were made portable with their context, and complex tasks were handed to a pre-warmed background model.
The costs are real. Continuous inference occupies the main model, instance switching and context compression briefly consume double compute, the relay adds an internal hop, and the dual-model system has to handle expired tasks and state divergence. OpenAI has disclosed the p95 figure and the round-trip reduction, but not the unit cost of continuous inference, real interruption accuracy, information loss after repeated compression in long sessions, or what proportion of background results arrive stale.
The lesson for anyone building agents is uncomfortable and useful. Real-time behaviour is not a gift from the model, it is a dividend from system scheduling, and the bottleneck is rarely GPU inference. It is usually some auxiliary component, a logging path or a state store, saturating first. OpenAI’s reframing was to stop asking how many tokens per second the GPU can push, and start asking how many frame-stable voice sessions the system can hold at once.
Editor’s note: this English report is an adapted translation of a Chinese-language original published by LeiPhone (leiphone.com). Figures, dates and direct quotations follow the source.