Crossing the Uncanny Valley of AI Voice

Copyright Vertical AI 2025. Written by Avi Santoso in August, 2025.

Remember when AI-generated images first started appearing? Most people would readily accept that a tree might look slightly off, but the moment they spotted a hand with six fingers, something felt deeply wrong. As humans, we have an incredible sensitivity to certain things. Nowhere is this more apparent than in voice conversations.

When building voice AI applications, there's a similar "six-finger moment" that breaks user trust: the awkward silence that happens when your AI needs to make a function call.

The Human Psychology of Voice Communication

In voice-only interactions, we're hardwired to expect continuous audio feedback. Think about the last time you called customer service. When the agent needs to look up some details, or process your data, they don't just mute themselves. They might say "please hold while I look that up," put you on hold music, or periodically reassure you with "hey, I know its been a while, but I'm still checking on that."

This isn't accidental. This is essential human communication. Without visual cues, we rely entirely on audio to understand what's happening. Remove that feedback, and users immediately start wondering: Did the call drop? Is the system broken? Did it hear me correctly?

Silence During Processing

Here's what happens by default in most voice AI systems: Your AI is having a conversation, determines it needs to call an external API (check account balance, book an appointment, look up inventory), makes that call, and then... nothing. Complete silence while the function executes.

From the user's perspective, they just asked a question and the AI vanished. They have no idea if:

The system is processing their request
The connection dropped
They need to repeat themselves
The AI is waiting for more information

This creates a terrible user experience, especially when function calls take more than a second or two. Users start filling the silence with "Hello? Are you there? Did you hear me?" breaking the conversation flow entirely.

The Prompt Engineering Trap

Many teams try to solve this by modifying their system prompts: "Always notify the user before making a function call" or "Keep the user informed during processing."

This approach has several critical flaws:

Non-deterministic behavior: You're relying on the LLM to remember to notify users every single time. Sometimes it will, sometimes it won't. Debugging why it works or doesn't work becomes a nightmare.
Wasted prompt space: You're using valuable tokens on notification instructions instead of domain-specific business logic that actually matters for your application.
Increased complexity: Every conversation now requires the AI to juggle notification logic alongside its primary tasks, making the system less reliable overall.

The other common approach is to simply accept the silence. Teams tell themselves "users will understand" or "it's only a few seconds." But this fundamentally misunderstands human psychology in voice interactions.

When someone can't see what's happening, even a 3-second silence feels like an eternity. Users lose confidence in the system and start second-guessing whether it's working correctly.

Our Advice

Study Real Human Patterns

The best starting point is listening to actual customer service calls. Notice how human agents handle these moments:

They set expectations upfront: "Let me check our system for you"
They provide time estimates: "This usually takes about 30 seconds"
They offer periodic updates: "Still looking into that for you"
Sometimes, you can hear them doing stuff: keyboard typing, mouth breathing, etc.
They use hold music for longer processes

Prioritize Deterministic Approaches

Unlike prompt-based solutions, deterministic approaches guarantee consistent behavior. When a function call is triggered, you can trigger for the agent to say something, play an audio, or mute the user. This makes the system predictable and fully testable with standard unit tests rather than expensive evaluation frameworks.

Match Strategy to Duration

Different wait times call for different approaches:

1-3 seconds: Brief acknowledgment ("Let me check that")
4-10 seconds: Verbal progress updates ("Still looking that up for you")
10+ seconds: Hold music or ambient audio with periodic verbal check-ins

Design for Interruption

Users will try to interrupt during processing—especially if it's taking longer than expected. Your system needs to handle scenarios like:

"This is taking a while, can you cancel that?"
"Are you still there?"
"Never mind, I'll call back later"

Decide upfront whether interruptions should cancel the function call, queue for later, or be acknowledged but not acted upon.

What Success Looks Like

When implemented correctly, asynchronous operation handling should feel completely natural. Users never wonder if the system is working. They always understand what's happening and approximately how long it will take.

The solution should be:

Fully deterministic: Testable with unit tests, not evaluation frameworks
Context-aware: Different responses for different types of operations
Interruption-friendly: Graceful handling of user input during processing
Transparent: Users always know what's happening and why

Most importantly, it should be invisible to users. They shouldn't think about the technical complexity. They should just experience smooth, natural conversation flow.

At our consulting practice in Perth, Western Australia, we specialize in building voice AI applications that feel natural and maintain user trust throughout complex interactions. If you're working on voice interfaces and want to discuss the technical and design challenges involved, we'd be happy to share our experience and explore how we might help with your specific use case.

Please contact us at hello@verticalai.com.au or visit our website at VerticalAI