Four leading real-time voice AI systems frequently acted on callers’ literal words while ignoring emotional cues in their voices, according to research that raises questions about their reliability in situations where tone and intent affect decisions.
The study, Real-Time Voice AI Hears but Does Not Listen, was submitted to arXiv on 24 June 2026 by Martijn Bartelds and Federico Bianchi of Together AI and James Zou of Together AI and Stanford University. The researchers tested OpenAI’s GPT-Realtime-2, Google’s Gemini 3.1 Flash Live, and Alibaba’s Qwen3.5 Omni Plus and Omni Flash Realtime.
The experiments examined three simulated telephone interactions in which the caller’s words conflicted with the way they sounded. In an emergency welfare callback, a caller insisted that everything was fine while crying. In a banking scenario, a customer authorised an $8,400 wire transfer in a frightened voice. In a recruitment call, a person agreed to volunteer for a food drive but delivered the agreement sarcastically.
All four systems closed the emergency callback as routine in all five runs under the crying condition. In the banking scenario, three systems approved the transfer in all five frightened-voice tests, while OpenAI’s model approved it in four of five. Every system enrolled the supposedly willing volunteer in all five sarcastic-voice tests, just as it did when the agreement was sincere.
The researchers then tested whether the models could identify the emotional cues when asked directly. Three of the four systems reliably recognised distress, fear or sarcasm in separate audio tests, despite often ignoring those cues when deciding what action to take. Alibaba’s Qwen3.5 Omni Flash showed weaker and less consistent recognition in some of the tests.
The distinction matters because recognising a speaker’s emotion is different from using that information to make a decision. A system may identify distress correctly but still treat a caller’s verbal reassurance as sufficient reason to end a conversation. The authors describe this disconnect between perception and action as an “emotional intelligence gap” in real-time voice AI.
The tests used speech synthesised with ElevenLabs, with five repeated runs for each main scenario condition. Five human listeners checked whether the intended vocal cues were audible. The findings therefore demonstrate a specific failure pattern under controlled conditions; they do not establish how frequently it occurs in real customer-service calls or prove that every voice AI product behaves the same way.
A separate evaluation published in the Proceedings of the 43rd International Conference on Machine Learning in 2026 found a wider gap between text-based reasoning and voice-agent task completion. Its τ-Voice benchmark covered 278 tasks involving multi-turn conversations, domain rules and interaction with external environments. GPT-5 in the benchmark’s reasoning-based text configuration achieved an 85% task-completion score, compared with 31% to 51% for voice agents under clean conditions and 26% to 38% under realistic conditions involving noise and diverse accents. The authors attributed 79% to 90% of observed failures to agent behaviour under their evaluation setup.
Attempts to correct the emotional-cue problem through explicit instructions produced mixed results in the first study. When instructed to consider vocal delivery and avoid acting on words alone, Gemini Live escalated the frightened transfer in all five runs, while GPT-Realtime-2 and Qwen3.5 Omni Plus did so in four of five runs each. Qwen3.5 Omni Flash escalated it in two of five runs. The emergency callback remained difficult: only one run across the four systems kept the line open to continue checking the crying caller’s safety under the tested instructions.
Discover more from Aree Blog
Subscribe now to keep reading and get access to the full archive.
