AI Voice Agent with n8n: The DIY Architecture with Twilio, Deepgram and ElevenLabs

Yes, you can build your own AI voice agent for under $0.10 per minute of call time — and you don't need a team of engineers to run it. The stack is Twilio for telephony, Deepgram to transcribe what the caller says, an LLM to decide the reply and ElevenLabs to speak back, with n8n orchestrating everything that happens around the conversation. It's the same architecture we use in client implementations, and in this article I'll show exactly how it goes together, the latency budget that separates a natural agent from a robotic one and — more importantly — when you shouldn't build it at all. If you haven't read our overview of AI voice agents for business, start there; this is the technical level.

What a real minute of conversation costs

Before architecture, the money, because most comparisons mix plan prices with per-minute rates and the result misleads. These are the public list prices of each piece (September 2026):

Adding the variable costs, each minute of conversation lands between $0.07 and $0.13; call it $0.10 per minute. A business doing 1,000 minutes a month (a clinic, a repair shop, a services firm) pays $80–130 monthly all-in. The market lands on the same number: the average cost of an AI voice agent is estimated at ~$0.08/min, versus $7.16 per human-handled call (CloudTalk, citing ElevenLabs/ContactBabel data).

And the opportunity side isn't cosmetic: in NextPhone's dataset of 1.4 million calls, 28.5% arrive outside business hours and 34.8% of those after-hours callers have buying intent. At 100 calls a month, that's ~10 buyers who hit voicemail.

The mistake of assuming n8n handles the audio

The most honest technical advice in this article: n8n is not the right place for the real-time audio loop. An AI voice-agent call has two planes that always get confused:

1. The media plane: audio flows from Twilio to Deepgram for transcription and from ElevenLabs back to the phone. Milliseconds matter here, and websockets rule.
2. The orchestration plane: what happens before, during and after the call — looking up the contact in the CRM, deciding whether to book or quote, updating the record, firing the follow-up.

n8n lives in the second plane, and that's where it shines. Teams that try to squeeze all the audio into n8n end up with fragile homemade websockets, workflow restarts mid-conversation and 3-4 second latencies that make callers hang up. The right sequence is to choose how the audio is handled first, then wire n8n to the call events.

The two build patterns

For the media plane there are two real paths, and the choice between them defines everything else:

Pattern 1, the simple one: Twilio uses a <Gather> TwiML verb with speech recognition; when the caller finishes speaking, Twilio sends the text to an n8n webhook; n8n asks the LLM and Twilio plays the answer via <Say> (ElevenLabs via API). It's literally a three-node n8n workflow and it works. The hidden cost is latency: each turn adds 1-2 seconds of round trip, and the conversation feels like talking over a radio. Acceptable for "new or existing customer?"; noticeable for "what do you need today?".

Pattern 2, the natural one: Twilio Media Streams opens a websocket to a small media server (Python or Node, ~200 lines) chaining Deepgram streaming, the LLM and ElevenLabs with bidirectional audio cutting. Callers can interrupt, the agent replies in under a second and the conversation flows. n8n stays out of that path: it listens to call-status webhooks, resolves customer context and executes actions. This is the pattern we use when the agent replaces a real receptionist.

Both share the same brain: the system prompt that defines the persona, tone and business rules. That part does live in n8n, and it's where you test it.

The latency budget: why your agent sounds robotic

Our clients' #1 complaint isn't that the agent says dumb things — it's that it takes too long. Latency compounds, and the human ear forgives very little:

Industry benchmarks (VAQI, cited in Deepgram's real-time voice agent guide) put the natural-sounding threshold below 300ms, acceptable between 300 and 600ms, and robotic above 1 second. The practical takeaway, which matches what we see in production: the LLM is the biggest and least predictable variable. ElevenLabs' own guide to building agents with Twilio says the same thing — the exact place where your latency budget dies is almost always the LLM.

Three rules we apply on every implementation:

1. Fast model, short context: a quick conversational model with a summarized history beats a giant model with full context. LLM latency of 300-500ms is worth more than an extra 5% of answer quality.
2. Stream, don't batch: if you wait for the sentence to finish before sending audio, you've already lost. Deepgram streaming emits partial hypotheses in ~150ms.
3. Trace every call: give each call a correlation ID and log stage timestamps (STT done, LLM first token, first audio chunk). When a client says "it sounds slow", the log tells you whether it was the LLM, the network or the audio queue. Without that trace, you'll be guessing.

n8n as the conductor: the call lifecycle

Once the audio is solved, the agent becomes an event workflow. The pattern we repeat for every client:

1. Inbound call: Twilio notifies n8n; the workflow looks up the number in the CRM (Clientify via API) and builds context: existing customer, lead or unknown?
2. During the conversation: n8n receives the transcript and detected intents and decides actions (book an appointment, send a quote, transfer).
3. Human transfer: golden rule — a keyword, three unresolved turns or an LLM failure triggers <Dial> to a human agent. Transfers are a system success, not a failure: in NextPhone's dataset, 73.8% of resolved calls routed correctly to their destination.
4. Wrap-up and follow-up: call summary into the CRM, task created for sales, and if relevant, continuation over WhatsApp with full context — the hybrid pattern we documented earlier.

Three production warnings learned the hard way: test AI-generated workflows in separate environments before touching production (the three-environment rule we cover in n8n AI assistant governance); define what happens when STT returns garbage (retry, transfer, or re-ask — never send empty text to the LLM); and watch per-call cost in month one, because context tokens inflate fast. If your agent will touch patient or auditable customer data, self-hosting the orchestration isn't optional — it's the project's main argument.

Build or buy: the decision almost nobody makes well

What "build your own voice agent" guides won't tell you: for most businesses, buying is the right answer. The SaaS agents we compared in AI voice platform comparison solve 80% of use cases for $50-400/month, with real Latin American Spanish (Fonema ships 200+ regional voices at <1.2s latency) and zero servers to touch. Building is rational only in a subset of cases:

In practice, the strongest business case we've seen is agencies: one build replicates across N clients, and the marginal cost per new agent is hours, not months. For a single business, the math almost never beats a SaaS.

What we learned building them

Closing with what you only learn by building: 80% of the effort isn't code, it's voice and script. The three fronts that eat the most time in real implementations are Spanish quality (regional accents wreck STT trained on neutral speech — test with real recordings from your market, not your own voice), interruption handling (a badly calibrated barge-in cuts callers off and frustrates them more than slowness) and monitoring (in month one, review transcript + summary + cost of every call, as you would for any new channel). If any of that sounds like maintenance you don't want, that's the strongest signal to buy. If you want full control, building your chatbot first is the best training ground: the voice agent shares the same brain — only the mouth and ears change. And if you'd rather have an experienced team build it, automation and chatbots is exactly the service we run every day.