Leaderboard — measured, not claimed
Every number on this page was measured by this platform with one stopwatch and one grader for all vendors. Column meanings, priorities and caveats: How it works.
What this is
One agent, four vendors, one stopwatch
We are choosing a voice-AI vendor between ElevenLabs, Deepgram, Vapi and Grok. Vendor marketing numbers are not comparable, so this platform builds the same car-sales agent on all four — same instructions, same greeting, same four tools, same AI model (gpt-4.1, temperature 0) wherever the vendor allows it — and runs the same recorded test calls against each one. Everything is measured by our own code, identically for every vendor, and saved. The leaderboard is the result.
Method
How one test call works
- The caller's lines become real speech. Each scripted line (e.g. "Hi, I'm looking for a used Toyota RAV4") is converted to audio once and reused, so every vendor hears byte-identical caller audio.
- A real call is opened to the vendor over the same live connection a customer's app or phone would use. No shortcuts, no text injection.
- The robot caller behaves like a person. It waits for the agent to finish speaking, then plays its line as a live microphone stream with real silences. The vendor must detect "the caller stopped talking" on its own.
- The stopwatch runs on every turn. Last moment of caller speech → first moment of agent speech. Same clock, same definition, every vendor. Vendor self-reported timings are stored but never used for scoring.
- Tools are really used. When the agent looks up inventory or books a test drive, it calls our HTTP API — the same endpoint, the same way, for all four vendors.
- The call is graded twice. Fixed pass/fail checks (right tool, right arguments, right facts spoken, hung up properly) plus an independent judge — Google's Gemini, which is not one of the vendors — scores the conversation 1–5.
To check what the agent said, we do not trust each vendor's own transcript. The agent's audio is transcribed by one independent transcriber for all vendors, so nobody is punished for their transcript API dropping words.
Reading the numbers
Every metric: meaning, computation, priority
| Metric | Plain meaning | How it is computed | Why it matters | Priority |
|---|---|---|---|---|
| Task success | Did the agent actually do the job? | Share of fixed checks passed per call: called the right tool with the right arguments, spoke the required facts (price, rate…), avoided forbidden tools, ended the call when told goodbye. 0–1, averaged. | A fast agent that books nothing is worthless. This is the closest thing to "would a customer have been served". | HIGH |
| Quality /5 | How good did the conversation feel? | Independent judge (Gemini, temperature 0) reads the transcript and scores accuracy, style rules, tool use, salesmanship and flow, 1–5. | Catches what checks can't: invented facts, robotic flow, calling a tool but never saying the result. | HIGH |
| Response p50 | Typical wait before the agent starts answering. | Per turn: caller's last speech audio sent → first agent audio back. p50 = the middle value over all measured turns (half faster, half slower). | Above ~1.5 s callers feel the lag; above ~3 s they talk over the agent or hang up. | HIGH |
| Response p90 / p95 | The bad moments. p95 = the wait only 1 turn in 20 is worse than. | Same per-turn values, pooled across all runs, 90th/95th percentile (nearest rank). | Customers remember the worst pause, not the average. A good p50 with a bad p95 means an agent that is usually fine and occasionally painful. | MEDIUM |
| $/min | What a minute of conversation costs. | Vendor's real bill where the vendor reports one (Vapi, ElevenLabs); our list-price model otherwise. Divided by conversation time, excluding test-harness padding. The "cost source" column says which. | Direct input to unit economics. Compare only bill-to-bill or estimate-to-estimate. | MEDIUM |
| Tool turn | Silence after the agent decides to look something up. | Tool call issued → agent audibly resumes. | Lookup-heavy flows live here. Caveat: Vapi speaks a filler ("one sec") before the result, which makes its number look better — read transcripts too. | CONTEXT |
| Greeting | Pickup → agent says hello. | Connection established → first greeting audio. | First impression only; happens once per call. | CONTEXT |
| Setup | Time to establish the connection itself. | Session requested → audio channel open (includes vendor API round-trips). | Engineering concern (capacity, retries); the caller mostly doesn't feel it. | CONTEXT |
| Runs / Turns | How much evidence sits behind the row. | Completed test calls, and total measured turns pooled for the percentiles. | Sample size. Don't act on a row with a handful of turns. | CONTEXT |
The dropdown
What the leaderboard views mean
| View | What it contains | Use it to answer |
|---|---|---|
| scripted, direct connection | The main benchmark: the robot caller over each vendor's own network connection. No phone network in the path. | "Which vendor's core product is faster / better / cheaper?" — the cleanest comparison. |
| scripted, over a real phone line | The same scripts, but the robot dials a real phone number through Twilio. Adds real phone-network delay. This is also how Grok's hosted agent is tested. | "What will a real caller on a real phone experience?" |
| live browser calls | A person talking from the Talk-to-an-agent tab with their microphone. | Hands-on impressions; demos. |
| live phone calls (in / out) | A person calling our Twilio number, or the agent calling a person. | End-to-end reality checks with a human in the loop. |
| every kind of test | All of the above mixed. | Totals only — never compare vendors here, the conditions differ. |
The leaderboard also filters by experiment: every run is stamped with a fingerprint of the exact agent setup and test scripts it ran under. Change the prompt, the model or a script and old results are kept but excluded — two different experiments are never averaged together.
Trust
What keeps the comparison fair — and the caveats
Held identical for every vendor
System prompt · greeting · the four tools and the code behind them (one shared HTTP API) · AI model gpt-4.1 at temperature 0 (where selectable) · the caller's audio, byte for byte · the stopwatch and its definition · the grader. A verification script reads the live configuration back from each vendor before every benchmark and refuses to run if anything drifted.
Differences we could not remove — keep them in mind
- Voices differ on purpose. Each vendor uses its own voice stack; that is part of what's being compared.
- Turn-detection: pinned to the same 500 ms where the vendor exposes a knob (Deepgram, Vapi); ElevenLabs and Grok use their own automatic detection — part of their latency is that choice.
- Vapi's built-in 0.4 s pause before speaking is turned off for measurement; add ~0.4 s to imagine an out-of-the-box Vapi.
- Grok is a different brain. It cannot run gpt-4.1 (it is its own speech-to-speech model), so its accuracy differences are model differences.
- Cost: Vapi and ElevenLabs rows are real bills; Deepgram and Grok are list-price estimates.
How to decide
Weigh Task success and Quality first — they say whether customers get served. Break ties with Response p50, then check p95 for consistency and $/min for economics. Use the phone-line view for what real callers will feel, and open a few transcripts in All runs before signing anything.
Run the identical scenarios against every vendor
Talk to an agent from your browser
Audio goes browser → this server → vendor, on the same path the automated tests use. The call is saved under the "live browser calls" view.
Inbound — call us
The Twilio number's Voice webhook points at
Then call from any phone. Pin a vendor per number with ?provider=vapi on the webhook URL.
Outbound — the agent calls you
Live call events
All runs — click any row for the full transcript
| When | Vendor | Scenario | View | Status | Dur s | Resp p50 | Worst turn | Tool ms | Task | Quality | Cost $ |
|---|
Run detail
Click a run above to see its transcript, checks and judge's reasoning.Loading spec…