Answers on the first ring · every hour

The calls you miss are the ones that were ready to buy.

A voice agent answers the phone when nobody can — after hours, during a rush, on the third simultaneous call. It understands what is being asked, looks the answer up in your systems, books the appointment, and passes anything beyond its remit to a person with the details already captured.

Voice is the hardest of the AI surfaces to build well, and the reason is not intelligence — it is time. A caller will tolerate roughly three quarters of a second of silence before they assume the line has dropped and start talking over you. Every architectural decision on this page follows from that one number.

Move over the waveform
+1 (416) 555-0148 inbound · 21:47 00:00
speaking ·
response barge-in on recording disclosed
Answers inUnder a second
InterruptionsHandled
Every callTranscribed
Anything unclearTransferred

The constraint

Six ways voice is harder than chat

Everything that makes a text assistant forgiving disappears on a phone line. These six differences drive every decision in the build.

01

There is no scroll-back

A caller cannot re-read the last answer. Anything long, numeric or spelled out has to be delivered in a way a person can hold in their head, or repeated on request.

In chat: the transcript sits there. On a call, it is gone the moment it is said.

02

Silence is a failure state

In chat, a two-second pause is nothing. On a call it reads as a dropped line. The system must acknowledge within a few hundred milliseconds even while it is still thinking.

The budget: roughly 750ms from end of speech to start of reply.

03

People interrupt

Callers cut in mid-sentence, and the agent has to stop immediately and listen. Anything that talks over an interruption feels exactly like the phone menus everyone resents.

Called: barge-in. Non-negotiable, and often skipped.

04

Audio is messy

Traffic, wind, hands-free kits, a bad line, accents, two people talking at once. Transcription quality varies enormously and the system has to behave sensibly when it is uncertain.

Result: confirm anything consequential before acting on it.

05

Names and numbers are brutal

Postcodes, order references, surnames, email addresses. Transcription is weakest exactly where accuracy matters most, so these get read back and confirmed rather than assumed.

Rule: never act on an unconfirmed reference number.

06

Everyone has been burned

Decades of phone menus have trained people to expect a fight. The first fifteen seconds decide whether a caller engages or starts jabbing zero — and the correct response to jabbing zero is to transfer them.

Design rule: say what it is, and offer a person early.

Interactive

The latency budget

A reply is not one operation. It is four, in sequence, and each one adds delay before the caller hears a word. Change the choices and watch the total move.

This is the whole engineering problem in one control panel. The most capable model is frequently the wrong choice, because a brilliant answer arriving two seconds late is worse on a phone call than a good answer arriving in half a second. The figures below are representative rather than guaranteed — real numbers depend on the provider, the region and the day.

Transcription
Reasoning model
Voice synthesis
network transcribe think speak
Network & telephony Speech to text Model response Text to speech

The stack

Seven layers between a ring and a reply

A voice agent touches more moving parts than any other AI surface — a carrier, a media server, two speech models, a language model, your business systems and a monitoring layer. Every one of them can add delay.

Telephony, SIP & media

Twilio Voice
Vonage
Aircall
Asterisk PBX
Kamailio SIP
WebRTC

Speech recognition & synthesis

Deepgram STT
ElevenLabs TTS
OpenAI Whisper & TTS
Google Speech
AWS Transcribe & Polly
FFmpeg audio

Reasoning & language

Anthropic Claude
Google Gemini
LangChain
Hugging Face
Ollama (self-hosted)
Python

Booking & business systems

Calendly
Google Calendar
Salesforce
HubSpot
Zendesk
Shopify orders

Handover & notification

Zoom
Google Meet
Jitsi
Slack alerts
WhatsApp follow-up
Telegram

Runtime

Node.js
FastAPI
Docker
Kubernetes
Redis (session state)
PostgreSQL

Applications

Six calls worth automating

All six are high-volume, low-variation, and currently either going to voicemail or interrupting somebody who was doing something else.

Call type 1 of 6
01 — Reception

After-hours answering

The call at nine in the evening that currently reaches voicemail and never calls back. Answered, understood, logged, and escalated if urgent.

  • Captures the reason while it is fresh
  • Urgent cases page the on-call person
  • Everything else waits, summarised

Usually pays for itself first

02 — Booking

Appointments and rescheduling

Checking real availability, holding the slot, confirming by text, and handling the reschedule three days later without anyone touching a calendar.

  • Reads live availability, not a guess
  • Confirms back before booking
  • Sends the reminder automatically

Clinics, trades, salons, studios

03 — Status

Order and job updates

Where is my delivery, has my part arrived, when is the engineer coming. High volume, entirely answerable from systems you already have.

  • Verifies identity before disclosing
  • Reads references back to confirm
  • Offers a text with the details

The biggest volume reducer

04 — Qualification

New enquiry triage

Working out what a caller needs, whether you serve it, and collecting the details a person would have asked for anyway before quoting.

  • Consistent questions, every time
  • Filters out-of-area enquiries politely
  • Warm cases transfer immediately

Stops good leads going cold

05 — Outbound

Reminders and confirmations

Appointment reminders, delivery windows, renewal notices — placed as calls, able to handle a question or a reschedule rather than just reading a script.

  • Respects calling hours and consent
  • Handles "actually, can we move it?"
  • Leaves a proper voicemail

Reduces no-shows measurably

06 — Overflow

The fourth simultaneous call

When everyone is already on the phone. Rather than ringing out, the call is answered, held usefully, and either resolved or queued with context.

  • No hold music with nobody behind it
  • Offers a callback with the slot booked
  • Scales to any number of lines

Peak periods and seasonal spikes

Honesty

Where voice agents genuinely struggle

This is the AI surface where the gap between a polished demo and a system that survives real callers is widest. Anyone who tells you otherwise has not run one at volume.

01

Heavy accents and poor lines

Transcription accuracy drops on accented speech, mobile calls from moving vehicles and older handsets. The mitigation is confirmation and a fast route to a person, not pretending it does not happen.

02

Spelling anything out

Surnames, street names, email addresses. Speech recognition is at its worst here. Reading back and confirming is slow, and it is still better than getting it wrong.

03

Upset callers

Somebody angry does not want a machine, however capable. Frustration should be detected in the first few seconds and transferred, not managed.

04

Background noise

Building sites, restaurants, cars with the window down. Noise suppression helps and does not solve it. Sometimes the honest answer is to offer a text conversation instead.

05

Long, winding explanations

Callers who tell the whole story before asking the question. Handling that gracefully — without cutting in — is genuinely difficult and needs real tuning.

06

Recording and consent

Ontario is one-party consent, but disclosure is still the right practice and other provinces and countries differ. Retention, storage and access all need deciding up front.

07

It costs more per interaction

Two speech models plus a language model plus carrier minutes. Still far cheaper than a missed call, and considerably more than a chat message.

08

Emergencies

If your callers might have an urgent or safety-critical problem, that path must be identified in seconds and routed to a human immediately. It is the first rule we write.

09

Sounding almost human

Voices are now good enough to unsettle people who were not told. We disclose within the first sentence — it costs nothing and prevents the reaction that follows finding out later.

Scope

What a voice agent build includes

More of this is telephony and testing than most people expect, and less of it is prompting.

01

Call audit

Listening to a real sample of your calls to find what repeats, how callers actually phrase things, and where the current process loses people.

02

Number and carrier setup

Provisioning or porting, routing rules, fallback to your existing line if anything fails. The phone must never simply stop working.

03

Conversation design

The opening line, the disclosure, how it asks, how it confirms, and how it exits. Written to be heard rather than read.

04

Speech tuning

Vocabulary hints for your product names and local place names, plus a voice and pace chosen to suit your callers rather than to impress.

05

System lookups

Bookings, orders, accounts and availability, with identity verification proportionate to what is being disclosed.

06

Barge-in and turn-taking

Interruption handling, silence thresholds, and what happens when both parties speak at once. This is where most of the tuning effort goes.

07

Transfer paths

Warm transfer with the summary already delivered, voicemail with a transcript, or a booked callback. Never a dead end.

08

Compliance

Disclosure wording, recording consent, retention periods and who can access transcripts. Agreed in writing before launch.

09

Load and failure testing

Simultaneous calls, provider outages, bad audio. What happens when the model is slow matters more here than anywhere else.

Questions

Voice agents, answered

Will callers know it is not a person?

Yes — it says so in the first sentence. Voices are good enough now that not disclosing is both misleading and counterproductive, because people react badly when they work it out later.

In practice callers are fine with it, provided the route to a person is obvious and quick.

How fast does it actually answer?

The target is under three quarters of a second from the caller finishing to the agent starting to speak. The latency budget above shows where that time goes and which choices spend it.

Hitting it consistently is the main engineering work — considerably more than getting the answers right.

Can it book into our actual calendar?

Yes, if the calendar or booking system has an API. It reads genuine availability, holds the slot during the call, confirms verbally and sends a text or email afterwards.

Reading live availability rather than approximating it is what makes the difference between useful and irritating.

What happens if it cannot understand someone?

It asks once more, and if it is still uncertain it transfers or takes a message rather than guessing. Repeatedly asking a caller to say something again is the fastest way to lose them.

Every one of those transfers is logged, so the pattern behind them can be fixed.

Do we keep our existing phone number?

Usually yes — either by porting the number or by forwarding to it. The setup is designed so that if anything in the stack fails, calls fall back to your current line.

The phone never simply stops working. That is a hard requirement, not a nice-to-have.

Is recording calls legal here?

Ontario operates under one-party consent, so recording a call you are part of is generally lawful. We disclose anyway, because it is better practice and because rules differ elsewhere.

Retention periods, storage location and who can access transcripts are decided with you and written down before launch.

How many calls can it take at once?

As many as you provision for. Unlike staff, capacity is a configuration setting, which is why seasonal peaks and unexpected spikes are where these systems earn their keep.

We load-test before launch rather than discovering the ceiling on your busiest day.

Can it call people, not just answer?

Yes — reminders, confirmations, follow-ups. Outbound carries more rules than inbound around consent and calling hours, and those get built in rather than bolted on.

We would not build anything resembling cold-call automation, for what it is worth.