Model and Inference Engineering · Staff
The caller interrupted the voice agent. Why did ASR hear silence?
Take a few minutes to form your approach. Then open a worked answer and compare the decisions.
Reveal a worked answer
Start with the audio path. The microphone picks up the caller plus the agent's voice coming out of the speaker. Acoustic echo cancellation uses a playback reference to remove that speaker contribution before voice activity detection and ASR. During an interruption, near-end speech and playback overlap. An echo suppressor that treats the caller's speech as echo can remove exactly the signal we need. A VAD that closes its gate while TTS is playing can create the same symptom without an AEC failure. Research on mobile full-duplex echo cancellation treats VAD and ASR downstream of echo cancellation and calls out hardware and reference-delay problems.
Record a consented diagnostic trace at four points with aligned timestamps: speaker reference, raw microphone, post-AEC microphone and the frames passed to ASR. If the caller is present in raw mic but missing after AEC, inspect playback-reference timing, echo-path changes, double-talk handling and suppressor strength. If the post-AEC signal has speech but ASR receives zeros or no frames, the gate or streaming transport is suspect. If ASR sees speech but no transcript, check language, chunking and recognition thresholds. Replay a set with caller speech at different playback levels, devices, rooms and interruption times. Do not tune an ASR model to repair frames it never received.
The real product decision is when to pause TTS. Detect an interruption using the earliest reliable near-end evidence, briefly duck or stop playback, and continue listening while the agent is speaking. That can improve the acoustic conditions, but a false trigger can cut the agent off. Measure missed barge-ins, false interruptions, detection delay and word error on overlapping speech. Also test what the user hears and whether a partially spoken assistant sentence is carried into the dialogue state. A beautiful AEC score on echo-only audio is weak evidence if users cannot interrupt the agent.
Continue reading
Related questions
Read beyond the question
Explore more model and inference engineering
Follow another question in this area, or search the complete Question Library.
Browse this area →Browse Question Library →