Training Voice Agents to Interrupt Properly

Most voice agents are built on a bedrock of clean, turn-taking audio, which creates a sharp friction point: when a user speaks while the model is outputting, the agent often treats that input as noise and continues its own monologue until finished. The DuplexDrama dataset addresses this by providing 800 hours of synthetic audio that incorporates natural overlaps, backchannels, and disfluencies. While the broader production process generated over 2,000 hours of total audio, this subset specifically targets the messy reality of multi-speaker environments where 3.8% of all turns carry at least one full-duplex behavior.

The Pipeline for Overlapping Audio

The complexity of this dataset relies on a four-stage synthesis pipeline. First, authors generate scripts that mirror the cadence of natural speech, including typical interruptions. Second, text-to-speech models render these individual speakers into distinct audio files. Third, an audio mixing stage calculates the precise timing for overlaps, such as when a listener inserts a short backchannel while the primary speaker continues. Finally, a validation stage ensures the mixing preserves emotional consistency and speaker identity.

Training for Concurrent Audio

To learn floor-sharing, a model requires frame-level annotations indicating speaker activity and state transitions. In a transformer-based architecture, these signals are processed by interleaving the input audio tokens with special speaker-turn markers. These markers serve as structural signposts, allowing the attention mechanism to distinguish between tokens originating from the agent's internal autoregressive generation and those representing the user's incoming stream. By concatenating the user's audio as a distinct sequence, the model can compute cross-attention weights between its own latent output state and the user's spectral energy, ensuring it doesn't conflate the two sources.

Before the gating mechanism can trigger a halt, the model must isolate interruption events. This is done through a classification layer trained on the specific spectral features of interruptions and backchannels—like sudden, short-duration increases in energy and characteristic pitch patterns—which are mapped to a dedicated set of 'interruption' tokens within the model's vocabulary. During training, the objective function uses these labels to penalize the model when it continues generating through a hard-stop cue. When the cumulative attention mass on these interruption tokens exceeds a predefined threshold, the decoder is gated to stop the autoregressive generation process, effectively treating the user's overlapping audio as a priority interrupt signal rather than background noise.

Consider an agent responding to a long list of instructions. If the user interrupts to clarify a point, the model's self-attention layers detect a high activation for the interruption tokens while still predicting the next token in the agent's monologue. Because the model has been trained to minimize the loss associated with concurrent signals, this high activation state forces the decoder to discard the remaining output tokens in the queue and switch from generation to listening mode.

The primary technical risk is the discrepancy between the synthetic training environment and real-world audio. Because the synthesis pipeline uses clean, individual speaker files to construct overlaps, the model does not encounter the specific reverberation profiles, background noise, or signal-to-noise ratio degradation inherent in live, non-scripted microphones. This creates a reliance on a clear, studio-mixed signal that may cause the model to fail when faced with the acoustic entropy of a real room. The next step for builders is determining how to layer synthetic interruption data over noisy, real-world audio to improve robustness.