The AI Voice Assistant That Answers To You, Not Your Customers

The useful version of this is a two minute call on the way to a meeting.

You say what happened. It tells you what is waiting. You approve three things out loud, park one, and hang up. By the time you park the car, the calendar entries exist and the handoff message has gone.

That is not a demo. It is the thing that finally made a queue of pending decisions drain instead of grow.

Two Products Share One Name

Search for an AI voice assistant and almost everything you find answers somebody else's phone. Books appointments. Qualifies leads. Handles support calls at 11pm. That is a real category and a good business, and our sister agency covers what it costs and when it is worth it in AI voice agents for business.

This is the other one.

It answers to you. It knows your queue, your calendar, your clients and your numbers. You ring it, not the other way round. And the test of whether it works is the other way round too: an outward-facing agent should need you as little as possible, while this one should need you at exactly one point, on purpose.

That flip is the whole architecture. An outward-facing agent works from a script and the worst case is an unhappy customer. This one sits on top of your real systems, so the worst case is it doing something on your behalf that you never agreed to.

What An AI Voice Assistant Looks Like When It Answers To You

Three properties, and none of them are about the voice.

It is a call, not a button. Always listening, interruptible, hands free. Push-to-talk is easier to build and it stops being useful the moment your hands are on a steering wheel.

The part that hears is deliberately stupid. A small worker owns the microphone and the speaker. It holds no model keys, no tools, and no memory of your business. It transcribes what you said, posts that text to one endpoint, and speaks whatever comes back.

Every decision happens somewhere else. One route owns the thinking, and the web app and your phone call the same route. Change how it thinks and you never touch audio code. Restart the audio and you cannot lose data, because it never had any.

That split is worth more than it sounds. A stolen credential from the audio process buys an attacker the ability to make it say things, not to make it do things.

The Wall That Makes It Safe

Here is the part most articles skip.

The model is handed no tool that can approve anything. Not told not to. There is no such tool, and asking for one does not produce it.

Your spoken "approve that" is matched by the route itself, before any model is consulted. The route works out which single item you meant from its own previous reply, refuses outright if you try to approve several at once, and then calls the identical function the on-screen approve button calls. Speech and tap mean the same thing because they run the same code.

The reason this matters is not distrust of the model. It is that the assistant's context is full of text other people wrote: meeting transcripts, client emails, task titles that came in from a chat app. If approval were a tool the model could call, then a carefully worded sentence inside somebody else's email becomes an instruction. Keeping approval out of the model's reach removes that entire class of problem rather than defending against it.

One live incident sharpened it further. Matching the approval word anywhere in a sentence meant "give me the urgent tasks you need me to approve" was read as an approval command. The word now has to lead the sentence. A question that merely contains it falls through.

The queue this all reads from, and the two autonomy lanes underneath it, are covered in agents propose, you decide.

Every Failure Sounds The Same

A voice call breaks differently from a web page. No error, no spinner, no stack trace. Just a person saying "hello?" into a phone.

Four separate causes produce that one symptom:

• A network blip. The browser drops and rejoins with a fresh identity. The default behaviour closes the session on the drop, so the rejoin finds an empty room.
• A busy machine. Frameworks mark a worker unavailable when the host's overall load is high, so unrelated work on the same box makes your assistant unreachable.
• A greeting spoken too early. Send speech before the browser has finished subscribing to the audio track and the line appears as a caption and is never heard.
• A sleeping screen. A phone that locks mid-sentence drops the microphone, which reads exactly like being hung up on.

None of these were found by reading code. Each one cost a call that went quiet in front of somebody, which is why each one is worth a test.

Silence Reads As Broken

People wait about two seconds before they say "hello?". An assistant answering from live business data sometimes needs longer than that, and pretending otherwise is how you end up with something nobody uses.

The instinct is that the plumbing is slow. Measured on this build, the tool server started cold in 0.17 seconds. Infrastructure was never the cost. It was model round trips.

So two things fixed it. The most-asked question moved out of a tool call and into the context that ships with every turn, which took it from timing out to answering in six seconds. Anything asked often belongs in context rather than behind a tool call, and that is a context engineering problem, not a voice one.

The second is that the worker speaks a short line when the answer is slow, then keeps waiting. That timer was set at 2.5 seconds first and it was wrong: every ordinary chat turn tripped it, so "how's it going" got "give me a sec, looking that up now". Needless, and a promise of a lookup that was not happening. Set above the measured chat turn and below the tool turn, it bridges real work and leaves normal conversation alone.

What It Costs

At published list prices in August 2026, a minute of conversation is roughly one cent for the room, three cents for speech out, half a cent for speech in, and a fraction of a cent for the model.

Speech out is the biggest line by a distance. Talking costs about five times what listening does.

That single fact should change how you write the prompt. An assistant that answers in two sentences is cheaper and better to use, which is a rare case of the nice thing and the cheap thing being the same thing.

Self-hosting removes the room cost entirely, since the media server and the agents framework are both open source. What you cannot remove is somewhere for it to run that never sleeps, which is a machine problem before it is a voice problem.

The Full Build

The guide is the whole thing: the worker that holds no keys, room tokens scoped to a microphone and a single call, why an unnamed worker answers other agents' calls, turn detection and the pre-roll padding that stops quick commands arriving as mush, the vocabulary hint list that echoed itself back as a turn nobody spoke, utterance merging, the ordered timeout chain, choosing a voice, spoken approval, the four silences, telephony against in-browser, and the per-minute costs.

Read the AI voice assistant guide

Four copy-paste prompts are included, one per piece.

This is one surface of a larger system. The screen version is the AI command centre, and the agents on the other end of the call are the subject of AI agent orchestration. If the phrase operating system is doing the heavy lifting for you, our sister agency has the business version in what is an AI operating system for business.

The Last Thing

Voice is the fastest way to reach a system you already trust.

It is the worst possible way to build trust you do not have yet. If the queue and the approval wall are not right, adding a microphone just means being wrong out loud, in the car, with your hands full.

Build the wall first. Then give it a mouth.

Take the IP Assessment