Skip to content

Voice conversation (WebSocket)

GET /admin/v1/chat/voice upgrades to a WebSocket that turns a user's speech into messages of an existing conversation. It is the live transcription relay plus a turn loop: the gateway detects when the speaker has finished a turn, stores the text as a normal user message and tells the client which message it stored. After storing a turn, the gateway answers it by speech on the same connection (see Spoken answer).

The flag that enables it stays off for every workspace until voice mode ships. The web app's client for it is described in Web app.

Endpoint and gates

Gate Result
no Upgrade: websocket header 400 { "code": "websocket_required" } (checked before the session, so an unauthenticated request without it is 400 too)
missing or invalid session, a viewer, an impersonated session 401 / refused, as for any admin route
a query string 400
Origin present and not the admin origin 403 origin_forbidden (an absent Origin, a non-browser client, is allowed)
workspace without voice_mode_enabled error not_entitled, close 4403

voice_mode_enabled is a tenant flag on PATCH /admin/v1/tenants/:id. Only a platform admin can change it; a tenant admin who tries gets 403 platform_admin_only and nothing is written (Tenants). Live transcription has its own flag and is independent of this one.

Start frame

The first text frame is the dictation start frame plus the conversation:

{"type":"start","encoding":"pcm_s16le","sample_rate":16000,"channels":1,"conversation_id":"<canonical uuid>"}
Start Result
valid, conversation writable by the caller ready follows
conversation_id absent, or any other key missing or extra (a language key included) invalid_start, close 1008
conversation_id present but malformed (not a string, not a canonical UUID, null) invalid_conversation_id, close 1008
conversation unknown, deleted, or another user's conversation_not_found, close 4404
conversation exists but cannot be written (read-only, not the driver of a live conversation) conversation_read_only, close 4409

The write gate is the same as for POST /admin/v1/conversations/:id/messages and runs before the speech service is dialled and again before every append. The language is the account's stored dictation language, never a key in the start frame. Only German, English and French are supported. If the speech service reports a partial or final transcript in another, well-formed language (for example it), that transcript is dropped and the session continues: it is not fed to the turn detector, not stored and gets no answer. A dropped final is answered with a notice frame (below); a dropped partial is silent. Words recognised earlier that were not yet a turn are not merged with later speech across the dropped utterance: the open hypothesis is closed, and it becomes a turn when its normal wait ends or at the end-of-session flush. A transcript whose language is absent or not a language code (for example de-DE) is still an upstream protocol violation and ends the session with internal_error (close 1011, "Voice mode failed.").

After start the client sends raw 16 kHz mono PCM16 as binary frames (even, non-zero length, at most 32000 bytes) and finally {"type":"stop"}.

Frames the gateway sends

Every message is a text frame.

  • {"type":"ready","language":"auto"|"<code>","limits":{"max_frame_bytes":32000,"idle_timeout_s":20,"max_session_s":600}}: audio may be sent from now on.
  • {"type":"partial"|"final","text":"…","lang":"…","t0":0.1,"t1":0.9}: interim and final transcripts, exactly as in live transcription.
  • {"type":"notice","code":"language_skipped","lang":"it"}: informational, the session continues. A final transcript in a language other than German, English or French was dropped (see above); lang is the language code reported. Clients that do not know a notice frame can ignore it.
  • {"type":"turn","message_id":"…","text":"…","lang":"…","reply_lang":"…","reason":"…"}: a turn was stored. message_id names the stored user message, and the frame is sent only after the message is stored, so it always names a row. reply_lang is the language the reply should be in (see below). reason is complete, incomplete_timeout, max_duration, overflow or flush.
  • {"type":"answer_start","answer_id":"…","turn_message_id":"…","lang":"…"}: the gateway starts answering the turn turn_message_id (see Spoken answer).
  • {"type":"audio","answer_id":"…","seq":N,"content_type":"audio/mpeg","bytes":N,"text":"…"}: one spoken sentence. Exactly one binary frame of bytes bytes follows immediately (at most 1 MiB; the content type is always audio/mpeg).
  • {"type":"answer_end","answer_id":"…","message_id":"…"|null,"status":"ok"|"error","code":"…"}: the answer is over. message_id names the stored assistant message (null when none was stored). On status ok there is no code key; with status error, code is one of the codes below.
  • {"type":"error","code":"…","message":"…"}: followed by a close with the matching code. message is fixed gateway text.

Refetch rule

The stored conversation is the truth. When a turn fails, when the connection closes with any code other than 1000, or when the client is unsure whether it saw every turn frame (a frame could not be delivered, a reload), it re-reads GET /admin/v1/conversations/:id and uses the stored messages. A turn the gateway stored but could not announce is still in the conversation.

Turn rules

  • A turn ends 700 ms after the last speech when the text is complete (HOLD), 2500 ms when it looks unfinished or a partial is still open (LONG_HOLD, reason incomplete_timeout).
  • Completeness is a language-aware check of the last sentence (a trailing conjunction, article, preposition or open clause is not complete). Only words that cannot end a sentence count: German zwischen is treated as unfinished, English between is not, because English can end a sentence on it ("What is it between?").
  • A turn never runs longer than 120 s (MAX_TURN, reason max_duration) and its text is capped at 16 KiB (reason overflow). max_duration is a bound, not a claim the speaker was interrupted: clients must not present it as a cut-off.
  • reply_lang is the language of the most recent speech final of the session (it persists across turns, so a turn that ends on a partial or on silence keeps the language of the earlier final). Only before the session has had any final does it fall back to the account's fixed dictation language, and when none is fixed (automatic) to the turn's own lang. The field is therefore always de, en or fr, never auto.
  • At the end of the session (stop, hang-up, upstream error or shutdown) the gateway first processes the transcripts already received, then flushes: the words spoken last become a final turn with reason flush and are never lost. The end phase is bounded to 10 s (3 s when the gateway is shutting down). A running spoken answer is stopped first; the short-lived playground token minted for it is then revoked in the background, off the session's clock; it is also revoked as soon as a later answer is refused by the access checks and when the conversation's gateway changes (skipped when the worker shuts down or the background revoke cannot be scheduled: the token then expires within its 30-minute lifetime).

Persistence

A turn is a normal user message of the conversation: same write gate, same storage, same history, search and export. It appears in GET /admin/v1/conversations/:id in speaking order. If a turn cannot be stored the session ends with an error close instead of a clean one, the text is not logged (only its byte length and the reason), and the client uses the refetch rule.

Spoken answer

After a turn frame the gateway answers it by speech, on the same connection. The answer is produced by the conversation's own model through the gateway's normal chat path (persona, policy, routing and billing all apply), then spoken sentence by sentence.

  1. answer_start, then for every sentence an audio frame (JSON) followed by exactly one binary frame with the MP3 bytes (at most 1 MiB, content_type audio/mpeg), in order. A client plays the binary frames in seq order and may start before the answer is complete.
  2. answer_end. Every answer, also a failed one, starts with answer_start and ends with exactly one answer_end, except when the session ends while an answer runs (stop, hang-up, session limit, idle timeout, upstream error, shutdown): then the answer is stopped, no answer_end is sent and the close frame ends it. Refetch the conversation for the stored partial answer (stopped_partial). The assistant message is stored as a normal message whose parent is the user message of the answered turn (turn_message_id). On status ok, message_id is the id of that message, but it is null on ok in these cases: the answer was spoken but could not be stored (a persist_failed event arrived after the text and no turn_committed followed), or the turn committed with nothing stored (an empty committed turn, or a conversation that is not persisted, turn_committed with message_id null); a message_id that is not a canonical UUID is also dropped to null. The client then simply has nothing to refetch.

One answer at a time. A turn that is stored while an answer is running is coalesced: only the latest pending turn is answered next, and the turns in between get no answer (they stay stored as user messages). The answer to the pending turn has that turn's message as its parent. Turns stored at the end of a session (the flush after stop, a hang-up or the session limit) get no answer.

Resumed turns. When the user goes on speaking before the answer is audible, the pending answer is cancelled and the stored turn is extended instead of a second turn being added, see Resuming a turn. Only one answer, the one to the last stored turn, can be interrupted this way.

The session continues after every answer_end, also after an error; the next turn is answered the same way.

Refetch. After answer_end with a message_id, the client re-reads GET /admin/v1/conversations/:id to get the assistant message; the stored row is the truth (see the refetch rule).

answer_end.code Fires when
answer_failed the model call failed, the connection to it broke, the history read used up the time budget, the conversation's model is gone (a model_unavailable status event) or has no local replacement (a no_default_available status event; nothing is spoken), the gateway refused the request (any error status not listed below, e.g. 401, 429, a policy 403), or there was an internal error while answering
answer_unavailable the conversation has no gateway or model, the user may not use the gateway, the history could not be read, the answer path is not configured, the conversation could not be read because of a storage fault, the token for the answer could not be minted, the gateway could not be read, or the answer thread could not be started
conversation_not_owned the speaker is not the owner of the conversation, or the conversation is missing or deleted (the read is owner-scoped and fails closed). Spoken answers are produced only in conversations the speaker owns; in a conversation shared with you through a project your turns are stored, but no answer is spoken (nothing is minted or called)
model_not_local the spoken turn could not be routed to a local Myra model. Causes: (1) the gateway has no enabled local Myra model; (2) the loopback call lacked a valid conversation binding (x-aig-conv-id) and so named a cloud model, which is not expected on the voice session's own call; (3) a transient routing fault while the local model was looked up (the turn is refused rather than sent to a cloud model; the next turn usually succeeds). A cloud conversation model alone does not cause this: the turn is answered by the local model, see Which model answers. In a Local only or tier-unresolved project the project's own refusal ends as answer_failed, see its row above. Nothing is spoken. Ask again; if it keeps happening, ask your administrator to enable a local Myra model for the workspace
conversation_read_only the answer could not be stored, because the conversation is read-only for this session (another client drives its live session)
speech_failed the text-to-speech service failed, or produced a clip larger than the 1 MiB frame limit. Audio already sent stays; speaking stops here. The text is still stored.
speech_busy the speech service is at capacity. Audio already sent stays; speaking stops here. The text is still stored.
speech_unsupported_language the reply language is not German, English or French (a safety check, not expected in practice: the speech recognition only reports those three languages). Nothing is spoken and no answer is stored (your turn is)
answer_too_long the text to speak exceeded the cumulative 64 KiB limit of the speech service. Sentences before the limit were already spoken (audio already sent stays; speaking stops here); the full text is still stored
interrupted the speaker went on talking before the first audio frame of the answer: the answer was cancelled and the turn is resumed (see Resuming a turn). Sent only when answer_start had already been sent; message_id is null and nothing was stored. Expected, not a failure: show no error
aborted the answer was stopped because the session ended (stop, hang-up, session limit, idle, upstream error, shutdown); never sent to the client; seen only in metrics and logs

If the client hangs up mid-answer the part generated so far is stored with stopped_partial set, and nothing keeps running: the next session opens without too_many_streams.

Answer bounds. The answer has a time budget of 420 s, counted from the start of the answer (the moment answer_start is sent); the read of the conversation history, the connection to the model and the first byte are included. The wait for the first byte is cut off at the time left of the budget (at most 120 s). Audio backpressure: while more than 2 MiB of audio waits to be sent to the client, speaking pauses until the client has read it. That wait counts against the same 420 s, measured from the start of the answer, both while the response is still being read and after it has ended; running out ends the answer with answer_failed. What is stored: when the budget runs out after the response has ended, the gateway has already read the whole response, so the full text is stored; when it runs out while the response is still being read, the gateway stops reading and closes its call to the model, so the part generated so far is stored with stopped_partial set (refetch the conversation). Speaking the remaining sentences after the response ends is bounded by the same 420 s and by the session lifetime; the budget is checked before every sentence is spoken and on every audio clip: once it is spent, speaking stops and the answer ends with answer_failed (the text is already stored in full). Further bounds: the model response body is capped at 4 MiB; and the gateway waits at most 120 s for the first byte and 120 s between chunks. Exceeding any of them ends the answer with answer_failed; each read from the model is itself cut off at the time left of the 420 s. answer_failed also fires when a single stream event exceeds 256 KiB, when the stream carries a terminal error (an error object, or a provider_error event: voice_model_not_local ends as model_not_local (this refusal normally arrives as a 403 before the stream opens, which ends as model_not_local; the in-stream mapping is a defensive fallback), live_not_driver as conversation_read_only, everything else as answer_failed), when the model is no longer available (see the answer_failed row above; neither status event is itself an answer_end code, while a request refused outright as voice_model_not_local ends as model_not_local), when the stream ends with [DONE] but carries no text and no stored answer (turn_committed), or when it ends without its [DONE] marker. A recoverable tool error in the middle of the answer (a tool_error status event) does not end it: the answer keeps speaking.

Which model answers

A spoken answer is always produced by the tenant's local Myra model, whatever model the conversation shows in the picker.

  • Local rule. When the conversation's model is a cloud model (or a provider-qualified pin such as provider/model), the gateway answers the spoken turn with the gateway's local model instead. A provider-qualified pin is overridden for the spoken turn; it does not produce a 409. The local model is the one the gateway would pick for the tenant among its enabled local Myra models.
  • The picker is untouched. The conversation's own model (model of the conversation) is not changed; typed messages in the same conversation keep using it.
  • The message row records the served model. The stored assistant message carries the local model that actually answered, not the picker model. The replaced model is kept in the request log of the turn (aig_voice_model_from).
  • Only a conversation-bound turn is swapped. The swap applies to the voice session's own loopback call, which always names the conversation with x-aig-conv-id. With a well-formed x-aig-conv-id the gateway replaces a cloud model with its local Auto pick (the replaced model is recorded as aig_voice_model_from in the request log meta). Without it a cloud model is not replaced but refused with 403 voice_model_not_local. The id is checked for format only; the swap only ever moves to this gateway's own local model. Ownership is not decided by the swap: the answer is stored only if the caller may write the conversation (its owner, or a shared-project member holding the live-driver lease). That check runs when the answer starts, and again when it is stored if no start claim exists. A transient fault while the local model is looked up refuses the turn the same way (403 voice_model_not_local) instead of sending it to a cloud model; the next turn is routed normally.
  • Refusal when there is none. If the turn cannot be routed to a local Myra model, it is refused and the answer ends with answer_end code model_not_local. Causes: the gateway has no enabled local Myra model; the request had no valid conversation binding and named a cloud model; or a transient routing fault while the local model was looked up (the next turn usually succeeds). In every case nothing is spoken, no upstream model call is made for that turn (the refusal is logged as a 403 without any upstream leg) and the session stays open.
  • Routing rules. A routing rule that names the original cloud model no longer fires for a spoken turn, because the turn is routed by the local model.
  • Projects. In a Local only or tier-unresolved project the project's own refusal still applies first (see the model_not_local row above).

What is spoken, and what is not

  • The sources footer of an answer is stored with the message but not spoken; neither are citation markers or file blocks. The text is also stripped of markdown before it is spoken.
  • A spoken answer is produced by the local (Myra) model: a spoken turn is routed only to the local Myra fleet, a cloud conversation model is replaced for the turn (see Which model answers) and non-local fallback models are removed. The check of the model that actually served each response against the Myra estate applies to Local only projects, as for typed messages. Without any enabled local model the turn ends with model_not_local, except in a Local only or tier-unresolved project, where the project's own refusal (model_not_allowed_for_project, conversation_project_unresolved, project_tier_unresolved; the related agent_area_unresolved applies to agent runs only) ends the answer as answer_failed.
  • Gateway tools: the gateway's own tools keep running on a spoken turn; only connectors (and connector setup) are left out. Web search is the exception to opt-in: it is not opted in on a spoken turn and runs only when the gateway's web search is set to mode always.
  • Memory: automatic memory capture is off on a spoken turn: the assistant is not asked to propose memories, because a proposal could not be shown for confirmation in a voice conversation. Stored memories are still used as on a typed turn.
  • MCP connectors: Connectors (MCP) are not used in a voice conversation; ask in a typed message to use them.
  • History replay: the model sees the stored conversation as message content only, bounded to the most recent 20 messages and 24,000 bytes, oldest dropped first. Both are defaults that the operator can override by environment (AIG_BRIDGE_MAX_TURNS, AIG_BRIDGE_MAX_CHARS).

x-aig-voice-lang

The gateway marks the spoken turn with the request header x-aig-voice-lang: <language> on its own loopback call. It is gateway-internal: not for browsers, and not in the CORS allow lists. The accepted values are de, en, fr, nl and da (at most 8 bytes; nl and da are accepted by the header but never sent by the voice session); de gets the German persona, every other accepted value the English one. A malformed value (empty, repeated, wrong case, unknown language, over-long) is refused with 400 invalid_voice_lang; an absent header is an ordinary turn. On its own (without a valid conversation binding) the header only restricts; with a binding the turn is answered by the local model. On a request that names its conversation (x-aig-conv-id, as the loopback call does) it switches the voice persona and the reply language, and a cloud model of that conversation-bound turn is replaced by the tenant's local model (Which model answers); the turn is refused with 403 voice_model_not_local when there is none (inside a Local only or tier-unresolved project the project's own refusal answers instead). On a request that carries the header but no conversation, nothing is replaced: it must name a local model itself, otherwise it is refused with 403 voice_model_not_local, and it is never anything more permissive. A request with this header is never served from or written to the response cache, and non-local fallback providers are removed from its route. On a voice turn the workspace capability block omits the voice / read-aloud line and reports connector setup as not available (the setup_connector tool is not offered on a spoken turn).

Resuming a turn before the answer is spoken

A turn can end too early: the speaker pauses mid-sentence, the 700 ms hold runs out and the turn is stored. If the speaker goes on before the first audio frame of the answer has been sent, the gateway takes the turn back instead of answering half a sentence:

  • Trigger: a partial or final transcript that contains at least one letter or digit (punctuation or an empty partial does not count) while the answer to the last stored turn has sent no audio. A speech event that arrives after the first audio frame does not cancel the answer; it is a normal new turn.
  • The answer is cancelled and voided: nothing it produces is stored or visible, and a tool it had not started does not run (a gateway tool already in flight at that moment finishes, its result is not shown). A client that had been sent answer_start receives answer_end with status: "error", code: "interrupted" and message_id: null; drop the audio of that answer_id that is still in flight.
  • The detector continues the same turn. At the next turn end the same stored user message is updated in place with the merged words (same message_id, no second row); the turn frame carries that id again with the full text. If the stored row no longer exists (turn_missing), the merged turn is appended as a new message with a new message_id. Any other update error ends the session fail-closed, like a turn that could not be stored; the merged words are never dropped silently. The merged turn is answered like any other, and may itself be resumed again.
  • The merged turn keeps the start time of the first part. The usual limits apply to the merged turn as a whole: 120 s and 16 KiB. A turn that would exceed a limit is not resumed, the running answer is left alone and the speech becomes a normal new turn.
  • A dropped unsupported-language utterance ends the merge window: speech after it never merges into the earlier turn, it starts a new turn (new message_id, no interrupted frame, the running answer is left alone). A resume that is already open is not undone by a dropped utterance.
  • Fail closed: if the turn cannot be taken back (limit), or the answer cannot be voided (storage fault), nothing changes and the answer keeps running, exactly as without this feature. One attempt is made per stored turn.
  • One database-heavy action per step: if the speech event is the very one that ends the turn (the turn is stored and answered in that step), it does not also take a turn back. Its words belong to the next turn, and the next speech event (the speech recognition sends a partial per update) takes that stored turn back while its answer still sends no audio.
  • A resumed turn is not answered speculatively; its answer starts at the merged turn end.

Speculative answer start

The session does not wait for the turn to end before it asks for the answer. While the end-of-turn detector holds a complete final (the 700 ms hold window), the session already starts the answer for that text. It sends a pre-allocated user message id as x-aig-parent-msg-id; that user row does not exist yet. Nothing is announced to the client while the answer is speculative: the answer events are held back.

  • The detector ends the turn with the same text and the same reply language: the turn frame carries the pre-allocated id, the user row is written under it, and the held answer_start, audio and answer_end follow. The answer is stored under that user message as usual.
  • The speculation is stopped only when different words are finalized (a new complete final with other text or reply language: the new text speculates at once), when the turn ends with different text or language, or when the user message could not be stored: then a normal answer starts after the turn. A partial that opens during the hold and is withdrawn or never completes does not stop it; the turn ends with the same text and confirms it, unless the speculative answer has already ended by then: the user message is then stored under a fresh id and a normal answer follows (not_confirmed; the ended speculation is billed like a Stop). A cancelled answer stores no assistant message, and an unconfirmed speculation never sends a frame.
  • The client hangs up inside the hold window: the speculation is cancelled; the held final is stored as a user message but is not answered.
  • An unconfirmed turn (the speculation was started but the user message was never written) ends with a parent_missing notice and a trace step and stores no message; it has no message id.

Billing. A cancelled speculation is not stopped at the model: it keeps generating at the gateway until its next failed write, and is metered and billed like a Stop (tokens and a request_log row).

Void. A barge-in (see Resuming a turn) writes the same marker row as an unconfirmed speculation, or soft-deletes the answer's user row when it is already stored; either way the pre-allocated id can no longer be used, nothing the cancelled answer stores becomes visible, and its gateway tools are refused by the check above. A tool that is already running when the void happens finishes; its result is discarded.

What a confirmed speculative answer does not have. While it streams it has no claim row. So a GET with await_turn_commit does not wait for it, turn_state does not show it as running in another tab, the reaper and the heartbeat do not apply to it, and a web Stop from another tab cannot reach it.

If the user message was never written when a speculative answer would be stored, the gateway stores no answer, writes one soft-deleted, empty marker row (never listed, never part of the history) and the turn ends without a message id. Any row under the same turn id, deleted or not, means the pre-allocated id is not used and the user message is stored under a fresh id.

A speculative answer runs no tool before its turn is stored. The first time the answer would execute a gateway tool (image or video generation, background removal, file writes and edits, office-file edits, the code interpreter, connector setup, file delivery, a gateway-side web search), the gateway checks, in one transaction on the conversation, that the pre-allocated user message exists. If it does, the tool runs as in any turn. If it does not, no tool runs, the turn ends there with the marker row described above, nothing else is stored, and the turn has no message id. A confirm that races this check is ordered by the conversation lock: the check first leaves the marker (the confirm then stores the user message under a fresh id and a normal answer follows), the confirm first lets the tools run. A web search the provider runs inside its own request is not a gateway tool and is not held back; it only reads.

Measured latency

Measured on a dev instance on 2026-10-30 with real speech recognition, the local model qwen3.8-27b and Piper (mp3): 40 answers in German and English (2 runs of 10 rounds in each language), 40 of 40 successful. The end of speech is the send time of the last voiced frame (RMS of at least 200); arrivals were polled every 5 ms:

Interval Median p95
end of speech to first audio 3.45 s 4.10 s (German 3.54 / 4.10, English 3.31 / 3.91)
end of speech to turn frame 1.94 s 2.65 s
answer_start to first audio 1.40 s 1.99 s

These are measurements of one setup, not guarantees.

Turn-end accuracy on real speech

The hold times were checked against real recordings, not only synthetic questions. The corpus is the VoxPopuli test split (European Parliament speeches, CC0-1.0): 240 recordings (German 73, English 87, French 80), each one speaker's uninterrupted utterance with natural pauses (an internal pause of at least 1.0 s), streamed in real time through the same speech-recognition leg as the voice session. Because nobody else speaks, every turn that ends before the last one is a false end (the turn was cut while the speaker was still talking). 36 synthetic questions (German and English, 600 to 2200 ms of silence inserted mid-sentence) were measured the same way.

Set Recordings False ends in the middle of a sentence after a finished sentence not alignable to the reference
Real, German 73 12 4 9
Real, English 87 22 11 2
Real, French 80 28 5 12
Real, total 240 62 20 23
Synthetic, total 36 17 0 0

With the resume of the previous section the early ends are mostly merged. Replaying the same recordings with a merge window (the first speech event within W ms after a turn end takes the turn back; the event that itself ended the turn does not count, the next one does), the false ends in the middle of a sentence fall from 79 (all 276 recordings, the 62 real ones in the table are a subset) to:

Merge window W 500 ms 800 ms 1200 ms 2000 ms
Recordings with fewer turns 60 81 91 97
False ends in the middle of a sentence left (of 79) 31 16 6 0
of these, real recordings (of 62) 24 12 5 0
of these, synthetic (of 17) 7 4 1 0

In the product the window is the time until the first audio frame of the answer, not a fixed number; it is longer than 800 ms in practice, because it includes the model's time to first token and the speech synthesis. The pins are in tests/fixtures/voice_turns/agf6047-corpus-merge-pins.json (W = 800).

On the real set the median time from the end of speech to the last turn frame is 2.1 s (p95 3.8 s).

Finding: a pause of more than about 1.5 s after a clause that looks complete can end the turn, because the 700 ms hold has already run out. The hold times (HOLD, LONG_HOLD, MAX_TURN) were deliberately not changed: shortening HOLD adds false ends, lengthening it delays every answer. A client that needs a different trade-off should not rely on the turn frame alone for long, deliberate pauses.

Limits: the recordings were captured on the dictation stream endpoint (/admin/v1/chat/transcribe/stream), not the voice-session endpoint (same relay leg and speech recognition); discard_hypothesis (dropped unsupported-language frames) is not modelled; the recordings are parliamentary speech, not questions to an assistant, and come from one speech-recognition model and one run. The turn times in the replay are modelled on the 30 ms frame grid, not measured. The recorded events and the expected turns of every recording are pinned in the repository (tests/fixtures/voice_turns/), and a test fails when a change moves any of them.

Errors and close codes

code Close Meaning
invalid_message 1008 not JSON, over-long, unknown or out-of-order message, a fragmented message
invalid_start 1008 start frame lacks a key (including conversation_id) or carries an extra one
invalid_conversation_id 1008 conversation_id present but not a canonical UUID
unsupported_format 1003 another encoding, rate or channel count
bad_frame 1007 empty or odd-length audio frame
frame_too_large 1009 audio frame over 32000 bytes
idle_timeout 4000 no start within 10 s, or no audio for 20 s. Only client audio resets the timer (pings and answer activity do not): keep sending audio, silence frames are fine, while an answer is played
session_limit 4001 the session reached 600 s; the last words are flushed and stored first
backlog_exceeded 4002 audio arrived faster than the speech service consumed it
not_entitled 4403 the workspace has no voice_mode_enabled
conversation_not_found 4404 the conversation is unknown, deleted (also mid-session, at the next turn) or not the caller's
conversation_read_only 4409 the conversation cannot be written to
unsupported_language 4422 the account's dictation language cannot be streamed
too_many_streams 4429 the user already has a voice or dictation stream open
busy 4013 the speech fleet is at capacity
unavailable 1013 speech service not configured, unreachable, dropped, or the gateway could not read or write the conversation
shutting_down 1001 the gateway is restarting
internal_error 1011 a gateway or upstream fault, including a turn that could not be stored and a transcript whose language is missing or malformed. A transcript in a well-formed but unsupported language is dropped, not an error
none 1000 normal end after stop

The shared close-code table and the relay's lifecycle details are in Live transcription.

Limits

  • 600 s per session, plus the bounded end phase; 20 s without client audio ends the session (pings and answer activity do not count: keep sending silence frames during playback); 32000 bytes per audio frame.
  • One slot per user, shared with dictation: a voice session and a live-dictation stream of the same user exclude each other (the second gets too_many_streams / 4429). A node also has a fixed global stream capacity (busy / 4013 beyond it).

Web app

The workspace chat uses this endpoint for its Voice mode button (see Chat: voice conversation). Client behaviour that other clients may copy:

  • The voice row names the model as Myra model.
  • The microphone and the audio player are created in the click handler, before any network call, because browsers only allow them after a user gesture.
  • While an answer plays, the client keeps sending silence frames (all-zero PCM) instead of microphone audio, so the session does not hit the 20 s idle timeout and the assistant does not hear itself.
  • Stop answer is local: the client stops playback, drops the audio frames of that answer that arrive later and goes back to listening. Nothing is sent and the socket stays open; the gateway still finishes and stores the answer.
  • The client re-reads the conversation (see the refetch rule) on answer_end, on a turn frame that names a message it does not have, and when the connection closes with a code other than 1000.
  • On answer_end with code: "interrupted" the client discards the audio of that answer, shows no error and keeps listening; the turn frame that follows carries the id of the same user message with the merged text, and the client replaces the text of the message it already shows.
  • A notice with a code the client does not know is ignored; language_skipped is shown as a short hint.
  • Close 4001 (session_limit) is the normal end of a session at the time limit and is shown as such; 4404, 4409 and 4403 are shown as errors.

Operators

aig_voice_session_outcome_total{outcome}, aig_voice_session_seconds, aig_voice_session_failover_total{result}, aig_voice_session_turns_total{reason} and aig_voice_session_notice_total{code} (code="language_skipped", once per dropped unsupported-language final; a dropped partial is not counted; each drop logs one WARN with the language code, never the text) on the internal /metrics endpoint, plus for the spoken answer aig_voice_answer_total{outcome} (ok or the answer_end code) and the histogram aig_voice_answer_first_audio_seconds (from the stored turn to the first audio frame; for a speculative answer it runs from the speculative dial inside the hold window, so it is shorter than the latency the user perceives; for a turn that was coalesced behind another answer, from the start of its own answer). No labels carry text, tenant or user. A barge-in adds aig_voice_resume_total{outcome} (merged: the stored turn was extended; refused_bounds: the turn could not be resumed because of a limit; void_failed: the answer could not be voided and keeps running), aig_voice_answer_total{outcome="interrupted"} (counted for a cancelled answer unless that answer had already ended on its own while the cancel was being stored, in which case its own outcome is the only count: one count per answer) and the histogram aig_voice_barge_in_seconds (the time of the cancel). The speculative answer start (see Speculative answer start) adds aig_voice_speculation_total{outcome} (confirmed, not_confirmed, cancelled, and dropped for a speculative start that never ran: the mint failed or the answer ended inside the spawn).