For operators who already run AI agents and now want to talk to them.
Build An AI Voice Assistant
That Answers To You
Not the one that answers your customers' phone. The one you call on the way to a meeting, that knows your business, tells you what needs deciding, and books nothing without your word. This is the full build: the room, the ears, the waiting, the mouth, and the four failures that make a live call go quiet.
A call, not a button
Always listening, interruptible, hands free
Ears apart from brain
The part that hears holds no keys and runs nothing
It cannot approve
No tool exists that lets it say yes on your behalf
Foundations
Two Kinds Of AI Voice Assistant
One answers your customers. One answers to you. They share a microphone and almost nothing else, and picking the wrong one costs you the whole architecture.
Search for an AI voice assistant and almost everything you find is the first kind. That is a real product category and a good business. It is not this one, and the two builds share almost nothing past the microphone.
Outward facing
Answers your customersPicks up the phone, books appointments, qualifies leads, deflects support calls. It is a front desk that never sleeps.
Success is calls handled without a human. It should need you as little as possible.
Inward facing
Answers to youKnows your queue, your calendar, your clients and your numbers. You call it. It reports, drafts, files, and asks you to decide things.
Success is decisions made faster. It should need you at exactly one point, on purpose.
The difference that changes the architecture: who the assistant is allowed to act for.
An outward-facing agent acts for the business against a script, and the risk is a bad customer experience. An inward-facing one sits on top of your real systems, so the risk is it doing something on your behalf that you never agreed to. Every design decision in this guide falls out of that second risk.
If the job in front of you is the first kind, stop here and read our sister agency on AI voice agents for business, which covers what they cost and when they are worth it. Everything below assumes you want the second kind.
Foundations
Split The Ears From The Brain
The process that hears holds no model keys, no tools and no memory of your business. Everything that decides anything lives somewhere else.
There are two jobs in a voice assistant and they want to live in different places. Hearing and speaking are real-time audio work in a long-lived process. Thinking is a request and a response. Put them in one process and every change to how it thinks becomes a change that can drop your call.
One turn, in order
You stop talking
Voice detection closes the turn. Pre-roll padding keeps the first syllable.
Fragments merge
A short window catches the rest of the sentence, so one thought posts as one turn.
Speech to text
Transcribed, with no vocabulary hint list attached.
One POST, bearer token
The only thing crossing the gap is a string of text. No keys, no tools, no state.
Deterministic ladder
Approval, undo, dispatch and capture are all decided before any model is asked.
The model, on a budget
Only reached if the ladder did not already answer. Hard time limit.
Markdown stripped
Or the mouth reads the asterisks out loud.
Spoken, plus captions
Captions stream to the screen. A named task spawns a card you can tap.
The worker holds
- →A speech-to-text key
- →A text-to-speech key
- →The address of one endpoint, and a bearer token for it
The worker never holds
- →Your model keys
- →Database access, or any tool at all
- →The prompt, the persona, or any memory of your business
What the split buys you
- →Restarting the audio process cannot corrupt anything, because it owns no data.
- →You can replace the brain entirely, including the model, without touching audio code.
- →The same brain endpoint serves voice, the web app and the phone, so all three behave identically.
- →A stolen worker credential buys an attacker the ability to say things, not to do things.
One trap worth naming early.
Agent frameworks assume they own the model. If you override that and point every utterance at your own endpoint, set the framework's model slot to a name that does not exist. Then the day someone removes your override, it fails loudly instead of quietly talking to a raw model with none of your rules attached.
Foundations
The Room, The Token And The Named Worker
Your server decides what a call may do, not the browser. And an unnamed worker joins every room on the server, including the calls meant for something else.
A live call is a room. Your browser joins it and publishes a microphone track, and a worker joins the same room to listen and answer. Everything about who is allowed to do what is decided at the moment the room token is minted, which means it is decided by your server.
Minutes, not days
The token is minted per call. If it leaks, it is already dead by the time anyone reads the log it leaked into.
Microphone only
No video, no data channel, no room admin. The browser is a microphone and a speaker. It has no other powers to steal.
One room, named
Never a wildcard. A token that can join any room is a token that can listen to any call.
Minted by the server
The browser asks for a call and the server decides what that call may do. The browser never holds a long-lived credential.
Name the worker, or two assistants will answer each other's calls.
A worker that registers without a name is dispatched automatically into every room on that server. Run a second voice assistant later and both now join every call, each talking over the other. Giving the worker a name switches it to explicit dispatch: it joins only when the token route asks for it by name. The cost of getting this wrong is that nobody joins at all, which is a much better failure than the wrong agent joining.
Fail closed on the way in
The worker reaches your brain endpoint with a bearer token. Write that check so a missing environment variable means the endpoint refuses everything, never accepts everything. Deployments lose environment variables far more often than they lose code, and an endpoint that opens up when its config goes missing is the worst possible shape for that mistake.
None of this works on plain HTTP. Browsers refuse microphone access and screen wake locks on insecure origins, so a real certificate is a hard requirement rather than a nicety. The machine and network side of that is covered in your AI operating system, on a machine that never sleeps.
Enjoying the guide?
Enter your details to unlock the remaining insights. We'll also send you the complete guide as a reference.
The Ears
Hearing: Turn Detection And The Pre-Roll Nobody Mentions
Is that speech, and have they finished? Get the second one wrong and you cut your own operator off mid-sentence, which is the fastest way to make a call feel broken.
Hearing is two questions asked constantly. Is that speech, and has the person finished? Get the first wrong and you transcribe the air conditioning. Get the second wrong and you interrupt your own operator mid-sentence, which is the single fastest way to make a call feel broken.
Three ways to decide the turn is over
Voice detection alone
Lowest latencyA small model decides only whether the audio contains speech. It is fast and cheap and it cannot tell a finished sentence from a thinking pause, so it will cut you off mid-thought.
A turn detection model
Best defaultReads the sound and the meaning together, so it waits when your sentence is clearly unfinished. Costs a little more time per turn and is worth it on anything conversational.
Push to talk
FallbackYou hold a button. Nothing to tune and nothing to get wrong, but it stops being hands free, which was most of the point.
The framework documents these and more, including how interruptions are detected, in its turn detection guide. Read it before you tune anything, because most of the knobs interact.
Pre-roll is the accuracy lever nobody mentions.
Short commands kept arriving at transcription as mush. The cause was the gate opening on the sound rather than before it, so the first consonant of a quick sentence was already gone. Raising the padding held ahead of detected speech from half a second to nearly a second fixed it, and the whole burst survived. It costs you that padding in latency and it is the cheapest accuracy you will ever buy.
Do not feed a vocabulary hint list to the transcriber.
It looks like free accuracy: hand it your client names and product words so it spells them properly. On short or quiet audio the model echoed that list straight back as though the user had said it, and a fake turn entered the conversation carrying words nobody spoke. In a system where speech can trigger actions, that is not a typo, it is an input you did not make. Pre-roll padding is the safe lever. Hint lists are not.
The detector itself can be small and local. The open-source one used here is Silero VAD, which is MIT licensed and processes a 30 millisecond chunk of audio in under a millisecond on a single CPU thread. Detection is not where your latency goes.
The Ears
One Thought, One Turn
People pause to think. Detection reads the pause as a full stop, and suddenly four replies are talking over each other, each answering a fragment.
People do not speak in clean sentences. They start, pause to think, and finish. Voice detection hears that pause and closes the turn, so one thought becomes two or three fragments. Each fragment then posts its own turn and races its own reply.
What that looks like live: four turns in nine seconds.
Four replies, all talking over each other, each answering a piece of a question nobody asked in that form. The assistant sounded confused. It was not confused. It was answering four questions correctly, and the questions were wrong because a pause had been read as a full stop.
The fix is a buffer with five guards on it
The merge window
Each fragment waits about a second for a successor. If one arrives, the newer arrival owns the merged turn and the older one goes quiet.
A generation counter
Two fragments waiting at once must not both wake up and post. A counter decides which one owns the buffer. Newest wins, every time.
The monologue flush
Past a character limit the buffer posts immediately. Somebody thinking out loud for a minute should not be held behind a window built for half sentences.
The stale drop
A fragment left behind by a cancelled turn must expire. Otherwise it surfaces inside a completely unrelated thought several minutes later.
A window of zero
Setting the window to zero disables the whole mechanism. Rollback is then a config change at 6pm, not a code revert.
Give people a way out mid-sentence.
"Never mind" and "cancel that" should match before anything else runs, empty the buffer, and bump the generation counter so a sleeping fragment does not wake up and post an empty turn. Walking away from a half-finished thought is normal in speech, and a system that cannot be abandoned mid-sentence is a system people speak to carefully. Careful speech is slow speech.
The Wait
The Latency Budget, And The Filler That Buys It
Silence reads as broken, not as thinking. Measure first: the plumbing was 0.17 seconds and the cost was somewhere else entirely.
On a phone call, silence does not read as thinking. It reads as broken. People wait about two seconds before they say "hello?", and the honest truth is that an assistant answering from live business data sometimes needs longer than that.
Measure before you tune
The instinct is that the plumbing is slow. It usually is not. Measured on this build, the tool server starts cold in 0.17 seconds. The real cost was model round trips: a turn answered from what the model already had ran around four to five seconds, and a turn that needed a database lookup ran seven or more. Infrastructure was never the problem, so tuning infrastructure would have fixed nothing.
The lever that worked was moving the most-asked question out of a tool call and into the context that ships with every turn. One question went from timing out to answering in six seconds with no tool calls at all. Anything asked often belongs in context, not behind a tool call.
Fill the gap out loud, and set the timer too high.
Race the reply against a timer. If the timer fires first, speak one short line and keep waiting. Two yields from one generator gives two spoken utterances, so the filler lands at once and the real answer follows behind it. It happens entirely in the worker, so it costs nothing but the words.
This was set at 2.5 seconds first and it was wrong. Every ordinary chat turn tripped it, so "how's it going" got "give me a sec, looking that up now": needless, and a promise of a lookup that was not happening. At 5.5 seconds, above the measured chat turn and below the tool turn, chat stays clean and only real work gets bridged. Keep the phrases neutral. "One sec" is honest whatever happens next. "Looking that up" is a claim.
Order the timeouts, and keep them ordered
The model's own budget
ShortestThe request timeout around it
LongerThe worker's HTTP timeout
LongestInvert any pair and the connection drops before the honest timeout line can be spoken. The user hears nothing at all, which is the exact failure the whole design is trying to avoid. A spoken "that is taking me too long" beats silence every single time.
Most of what makes a turn slow is what the assistant has to go and find out. Doing that thinking once, at the point the information arrives rather than at the point you ask, is the difference between an assistant that gets faster and one that does not. That pattern is in context engineering.
The Mouth
Speaking Without Embarrassing Yourself
Four ways the simplest part of the stack goes wrong in front of someone, including the one where it reads the asterisks out loud.
The mouth is the simplest part of the stack and it has the most ways to embarrass you. All four of these were found live, in front of someone.
Strip the markdown first
Models write bold, headings and bullets. Speech synthesis reads them literally, so a formatted answer comes out as "asterisk asterisk Execute". Strip formatting on the way to the mouth and turn line breaks into sentence pauses. Take out the marks, never the words.
Pick the fast model, not the pretty one
Voice quality matters far less than the gap before it starts. A low-latency synthesis model is the right default for a live call, and the richer models belong in anything pre-recorded where nobody is waiting.
Choose a voice you can stand daily
You will hear this voice several times a day for months. Test the shortlist on your own sentences, not the vendor's demo script, and test it through the speaker you will actually use. Phone speakers flatten a voice in ways headphones hide.
Wait a beat before the greeting
Speech sent before the browser has finished subscribing to the agent's audio track lands as captions only. The line is on screen and nothing is heard, which reads as a dead assistant. A short pause after joining fixes it.
How fast is fast enough?
The vendor used here publishes roughly 75 milliseconds for its fastest synthesis model, against roughly 280 for its conversational one, and both figures exclude your own application and network time. Against a turn that takes four seconds to think, that difference is small. It stops being small the moment the answer is already in hand and the only thing left is saying it.
Never tell it to say "let me look into that".
A prompt once did, and the assistant replied with only that line and ended the turn. The whole turn was wasted and the question was never answered. The worker's filler already covers the wait. The model's job is to answer, or to go and find out and answer, in the same turn.
The Wall
It Cannot Approve Anything, By Construction
Not instructed not to. There is no approve tool, and asking for one does not produce it. Your own words are matched before any model is consulted.
This is the part that makes an inward-facing assistant safe to give real access to, and it is one sentence: the model is handed no tool that can approve anything. Not instructed not to. There is no such tool, and asking for one does not produce it.
Approval is recognised, not generated.
Your spoken "approve that" is matched by the route itself, before any model is consulted, and it then calls the identical function the on-screen approve button calls. Speech and tap mean exactly the same thing because they run the same code. The model is never anywhere in that path, so no amount of clever wording in an email, a meeting transcript or a task title can talk its way into an approval.
One item, read out loud first
The target is resolved from the single task named in the assistant's previous reply. If that reply named two, neither can be approved. You cannot approve something you were never told about.
The approval word has to lead
Matching anywhere in the sentence meant "give me the urgent tasks you need me to approve" was read as an approval command. A question that merely contains the word must fall through to the brain.
Bulk is refused outright
"Approve all of those" is the one sentence where a mishearing is unrecoverable. It is not interpreted, it is declined.
Undo refuses once it is real
While nothing has left the building, undo reopens it. Once the calendar entry exists or the message has gone, flipping the status back would only make your board lie about the world.
The spoken numbers and the on-screen cards must share one module.
"Task three" has to mean the third card, always. If the list the assistant reads from and the list the screen renders from are two pieces of code, they will drift, and the card you are looking at stops being the row that gets approved. This is not tidiness. It is the difference between a confirmation and a guess.
Voice is one surface onto a queue that the web app and your phone also work from. The queue itself, the two autonomy lanes, and the executor that applies your decisions are the subject of the AI command centre guide, and the short version of why it works that way is in agents propose, you decide.
Operations
Four Ways A Live Call Goes Quiet
No error, no spinner, no stack trace. Just a person saying hello into a phone. Every one of these produces that same symptom from a different cause.
A voice call fails differently from a web page. There is no error message, no spinner and no stack trace. There is just a person saying "hello?" into a phone. Every failure below produces that same symptom, and each has a different cause.
You rejoin and nobody is there
SilenceCause. A network blip disconnects the browser and it rejoins with a fresh participant identity.
Fix. The default behaviour closes the session on that disconnect, so the rejoin finds an empty room. Keep the session alive across reconnects and let job shutdown do the tearing down.
You join to silence, every time
SilenceCause. The framework marks a worker unavailable when the host machine's overall load is high.
Fix. On a shared box, unrelated work makes your assistant unreachable. If this worker takes one call at a time and its own work is small, raise the threshold so it always accepts the job.
You can see the reply but not hear it
SilenceCause. Speech sent before the browser finished subscribing to the agent's audio track.
Fix. Captions render and the audio never plays. Wait a beat after joining before the first line.
The screen sleeps and it hangs up
SilenceCause. A phone that locks mid-sentence drops the microphone.
Fix. Hold a screen wake lock for the duration of the call, and take it again when the tab regains focus. Browsers drop the lock when you switch away.
An unreachable brain must still say something.
If the endpoint is down or answers with an error, do not let the exception travel upward. Catch it and return a short spoken line: it cannot reach its brain, try again in a moment. The mouth has to produce words no matter what happened behind it, because the alternative is a caller who does not know whether the system is thinking or dead.
Write a test for each one
None of these are hypotheticals and none were found by reading the code. Every one cost a call that went quiet in front of somebody. That makes them worth a test each, and a test suite of real incidents is worth more than a test suite of imagined ones. The approval rules in particular should keep their own copy of the patterns they check, so any drift in the route shows up immediately as a failure rather than as a bad approval.
Operations
In The Browser, Or On A Real Phone Line
Less a technical choice than a question of whether you want a screen in the loop. Taking the screen away costs more than it looks like on paper.
There are two ways to reach an assistant by voice, and the choice is less about technology than about whether you want a screen in the loop.
In the browser
What this build usesOpen a page on your phone and talk. No phone number, no per-minute carrier cost, and the screen is right there for captions and tappable cards. It needs a real certificate, and it needs you to open something first.
A real phone number
Add later, if everDial in from any phone, including in the car with no screen. You add a telephony layer, a number, per-minute carrier charges, and narrower audio that makes transcription harder. You also lose the screen, and with it the cards.
Losing the screen costs more than it looks on paper.
Spoken approval works because you heard the item read out and you can see the card while you say yes. Take the screen away and the only record of what you just approved is your memory of a sentence. That is a real downgrade in a system where approval sets off work in the world, so a phone number is a genuine feature and not a free one.
The same framework handles both, so this is not a rebuild. Its telephony documentation covers connecting a number to the same agent you already have. Build the browser version first, live with it for a fortnight, and then decide whether the car journeys justify the number.
Operations
What It Actually Costs To Run
Four billing lines, and almost everyone guesses the wrong one is biggest. The answer should change how you write the prompt.
A voice assistant bills in four places, and people almost always guess the wrong one is biggest. These are list prices per minute in US dollars, published by the framework vendor and checked on 16 August 2026. Treat them as calibration rather than a quote.
Transport, the room itself
$0.0100
Removed entirely if you self-host
Speech out
$0.0300
The biggest line, by a distance
Speech in
$0.0058
Varies by provider
The model
$0.0014
Varies by model, and by how much you ask of it
Source: the framework's published pricing. The free tier includes 1,000 agent session minutes, which is more than enough to find out whether you will use this at all.
Talking costs about five times more than listening.
That single fact should change how you write the prompt. An assistant that answers in two sentences is faster to listen to and a lot cheaper than one that reads you a summary you did not ask for. Tell it to be brief and you have cut your largest line item while making it better to use. Those two rarely point the same way.
Two things that change the sum
- →Self-hosting. The media server and the agents framework are both open source and free to run on your own machine, which removes the session line and leaves you paying only the speech and model vendors.
- →Going direct to the speech vendor. Bought straight from the source, speech is billed per character rather than per minute, so the cost tracks how much it says rather than how long you are on the call. A long silence is then free, and a long answer is not.
For the kind of use this build is for, a handful of calls a day between meetings, the whole thing lands well under what most operators spend on a single seat of project management software. The cost that matters is your attention, not the per-minute rate.
VoiceVoiceisisthethefastestfastestwaywaytotoreachreachaasystemsystemyouyoualreadyalreadytrust,trust,andandthetheworstworstpossiblepossiblewaywaytotobuildbuildtrusttrustyouyoudodonotnothavehaveyet.yet.GetGetthethequeuequeueandandthetheapprovalapprovalwallwallrightrightfirst.first.ThenThengivegiveititaamouth,mouth,andandthethewholewholethingthingcollapsescollapsesintointoaatwotwominuteminutecallcallononthethewaywaytotoaameeting.meeting.
Build it in this order
The brain endpoint, answering with plain text and nothing else. Test it with a terminal, no audio anywhere.
The deterministic ladder in front of it. Approval, undo and dispatch decided before any model is asked.
The worker, joining a room and speaking whatever the endpoint returns.
Coalescing and the filler line, once you have measured your own turn times.
The resilience pass, with a test for each failure.
Every step is useful on its own. If you stop after the second one you still have an assistant, you just type at it.
Jump back to a section
If you are earlier than this and still deciding whether a live voice model belongs anywhere near your operation, the honest version of that question is in what real-time voice models changed.
Read next
Primary sources