Nobody talks to AI the way they type
I built a voice assistant into an app. Most of what I learned was about how people talk.
I built a prototype voice assistant into an app, on Gemini. You press a button, say what you want, and it does it in the app for you.
Adding voice to an app has never been easier. The models listen and answer in real time, and the voices sound human. The hard part is somewhere else.
Once people started talking to it, most of what I learned wasn't about AI. It was about how people talk.
Typing is editing. Talking isn't.
When you type to a chatbot, you edit before you send. You think, write, delete, rewrite. By the time you press enter, the computer gets your final draft.
When you talk, it gets your first draft. You start a sentence before you know how it ends. You say "um". You start over halfway through. You correct yourself: "no, wait, the other one." People who study conversation find a stumble like that in roughly one word out of every twenty.
People also say more. Talking is about three times faster than typing on a phone, so words are cheap. A typed request is a few keywords. A spoken one is a little story with the request somewhere in the middle, sometimes two requests in one breath.
And some people don't know what to say at all. A text box at least looks like a search bar. A microphone button gives no hint. People freeze, or say "hello?", or ask what it can do.
Most AI products are built and tested by typing. The prompts are typed, the test cases are typed, and the people building it type all day. Voice is a different input. You're not getting a question. You're getting someone thinking out loud.
"That one"
People point. "Open that one." "Delete this." "The second one. No, the one below it."
Between two people in the same room, that works, because each can see what the other is looking at. An assistant inside an app is in the same room. It should see what you see.
So the assistant needs to know what's on the screen: which page is open, what's selected, what's in the list. Without that, half of what people say means nothing. With it, "that one" is the easiest request in the world.
It should answer on the screen too. Talking is a fast way in and a slow way out. You can skim a list with your eyes, but you have to listen to a spoken one from start to finish. Let the voice take the request and let the screen show the result.
Show what you heard
Messy input means the assistant will often get it slightly wrong. Not wildly wrong. Slightly. The right meeting, the wrong day. The right person, the wrong action.
So don't act on a guess. Show what you understood and let people fix it before anything happens: "Move the team meeting to Thursday at three?" Saying yes takes a second. Untangling the wrong action afterwards takes much longer, and it costs the trust you need for people to try a second time.
When a request is too vague to guess, ask one short question. Not three. An assistant that interrogates you is worse than a form.
A human voice is a promise
The more human it sounds, the more people treat it like a human. They expect it to remember what they said a minute ago, get the joke, and know what they meant rather than what they said. On my prototype, that's exactly what happened: people expected it to behave like a person.
A text box sets the bar at a search engine. A warm voice sets it at a person. Better voices make this worse, not better.
There's a second problem. A screen shows you its limits. Every option has a button, and anything without a button doesn't exist. A voice hides its limits. You can ask it anything, so people do, including things the app was never built for.
Demos push expectations even higher. In a demo, the person talking is the person who built it, saying things they know it handles. Everyone watching sets their expectations by that.
So the voice has to be honest about what the app can do. Say it up front. Give people a few things to try the first time. When someone asks for something it can't do, say so plainly and suggest something it can. Don't let the voice promise more than the app behind it can deliver.
The misses are the roadmap
Every request the assistant can't handle is a person telling you, in their own words, what they wanted from your app. Nobody fills in a feedback form that honestly.
Keep those requests, with permission, and read them every week. Some are phrasings you didn't expect for things the app already does, so teach the assistant to understand them. Some are features nobody asked for in any meeting. That's your roadmap.
Search boxes did this for a generation of websites. Voice does it with more words and less filtering.
Nobody can tell you if it's good
With a button, it either worked or it didn't. With a conversation, what's the right answer to "can you sort out the thing from earlier"? Ask which thing? Guess? Say it can't? Each one is defensible.
On the prototype, there was no clear way to measure whether it worked, or whether a change made it better. Without a measure, you judge by feel. You try it, it seems better, you move on.
That doesn't scale, and it lies to you. You built it, so you know what it understands, and without noticing you talk to it the way it likes. The builder is the worst tester it has.
The fix is deciding what good means before you build. Collect real requests from people who didn't build it, recorded the way they actually said them, messy. Write down the right outcome for each. Score every change against that list. Score whether it did the right thing, not whether it sounded nice.
Then watch the signals people give without being asked. How often do they repeat themselves? How often do they say "no"? How often do they give up and tap the button instead? A person reaching for the screen is the most honest review a voice assistant will ever get.
What I'd do next time
- Test with people who didn't build it, from day one. The team talks like the prompt.
- Give the assistant the screen: what's open, what's selected, what's in view.
- Show what it understood, and let people correct it before it acts.
- When a request is vague, ask one short question. Don't guess, and don't interrogate.
- Give people a few things to say the first time they open it.
- Make the voice as human as the product is capable of, and no more.
- Keep the requests it couldn't handle, and read them every week.
- Collect real requests and score against them. Feel isn't a metric.
Building it taught me less about AI than about people. You type what you mean. You say what you're thinking. Build for the second one.
Sources
- Comparing speech and keyboard text entry for short messages in two languages on touchscreen phones (Ruan et al., Stanford, 2016)
- Disfluency rates in conversation: effects of age, relationship, topic, role, and gender (Bortfeld et al., Language and Speech, 2001)