Voice Agent Evaluation: Test the Call, Not the Transcript

Amazon made Nova 2.5 Sonic generally available on October 5 with improved reasoning, instruction following, tool-calling accuracy and lower latency. The release also carries forward controllable turn-taking, asynchronous tool calling, voice and text in one session, seven languages and a 256K context window.

Those capabilities make richer calls possible. They do not make a transcript a sufficient acceptance record.

A spoken appointment time can differ from the generated text. A tool can return success while the booking system holds a different state. An interrupted request can leave an older tool call running. A polished escalation can still reach no human. Each failure crosses a different evidence plane, so one score cannot decide release.

The practical unit is a voice run receipt: one scenario, one call, one expected business state, the audio and timing evidence, every tool result, an authoritative read-back and the person or system that owns the terminal state.

Nova 2.5 Sonic changes capability, not accountability

AWS's Nova 2.5 Sonic announcement describes a speech-to-speech model that can keep context across multi-step work and use connected tools during a live conversation. Its example moves from looking up an order to checking return eligibility and initiating a return.

That sequence contains at least three claims:

1. The agent understood the caller's intended order and action.
2. The tool invocation matched the authorized action.
3. The system of record now contains the intended result.

The model can help with the first two. The application still has to prove the third. It must also prove that the caller heard the correct order number, that interruptions did not corrupt the request and that control ended with either the agent or an acknowledged person.

AWS makes the boundary visible in its current voice travel concierge architecture. Authentication, the agent runtime, MCP tools, business APIs, records, policy retrieval and live-agent escalation are separate components. The caller experiences one conversation, but the operator has to join evidence from all of them.

Do not ask whether the voice model is good enough in the abstract. Ask whether one named workflow can reach only accepted terminals under the conditions real callers create.

Use five release layers

A useful voice agent evaluation has five ordered layers:

1. Simulation validity: Did the test caller follow the assigned goal and did the conversation end in a way that can be scored?
2. Task accuracy: Did the agent follow policy, call valid tools, preserve key spoken entities and leave the expected business state?
3. Spoken experience: Did turn-taking, concision, progression, interruption recovery and silence remain inside the workflow's declared contract?
4. Effect reconciliation: If the agent wrote something, does an authoritative read-back agree with the requested action and provider receipt?
5. Terminal ownership: Did control return to the agent, reach a safe stop or move to an acknowledged human?

The order matters. A broken simulation should not become a model failure. A smooth call with a wrong business state should not pass because its latency score was good. A correct write without read-back remains uncertain. A handoff without acknowledgment remains unowned.

ServiceNow's October 2026 EVA-Bench research supports this separation. It validates bot-to-bot simulations before scoring, then distinguishes accuracy from experience across complete spoken conversations. Its public suite contains 213 scenarios across three enterprise domains and tests accent and noise perturbations.

The point is not to import a vendor's composite score as your release decision. It is to preserve the distinction between correctness and experience, then bind both to the business effect your workflow owns.

Validate the test before judging the agent

Synthetic callers are useful because they can run the same scenario repeatedly, vary a persona and introduce noise without asking a human to repeat a call all day. They can also break the evaluation.

If the simulator forgets the assigned account number, accepts an option the persona should refuse or ends before the terminal action, the resulting score says little about the agent. Mark that run invalid and repeat it. Do not average it into a failure rate.

EVA's public scenario structure binds four things together:

• the caller's exact goal;
• the caller's persona and behavior;
• the scenario database the tools will query or change;
• the expected final database state.

That structure makes versions comparable. The same caller goal and starting state can be replayed against a prompt change, a model change or a new turn detector. Without a stable starting state, a better score may only reflect an easier scenario.

Keep test validation outside the agent's score. A minimum run record should say whether the simulator stayed on task, whether the agent reached a supported end condition and whether the audio files, timestamps and tool logs are complete. Only then should the run enter the release set.

Separate task accuracy from spoken experience

A voice system has two outputs at once: the action it takes and the conversation a caller experiences. Both can fail independently.

For task accuracy, inspect:

• the expected final business state versus an authoritative read-back;
• tool name, parameters, authorization and idempotency key;
• policy faithfulness;
• important entities such as dates, names, quantities and confirmation codes in the actual audio;
• the final disposition and owner.

For spoken experience, inspect:

• whether the system entered and yielded turns correctly;
• the delay after the caller finishes speaking;
• dead air during tool work;
• whether responses are speakable and concise;
• whether the conversation keeps moving rather than repeating itself;
• recovery after barge-in, correction and background noise.

EVA's open metrics documentation keeps task completion, faithfulness and spoken fidelity in its accuracy layer. It treats turn-taking, conciseness and conversation progression as experience. Diagnostics such as response speed, word error rate, tool-call validity and key-entity transcription help explain why a run failed.

That is a better operating shape than one quality score. A workflow owner can block a wrong state immediately, while an engineer can use diagnostics to locate whether the fault sits in speech recognition, turn detection, the model, a tool or synthesis.

Choose thresholds from the workflow, not from a generic benchmark. A caller waiting for a policy lookup and a caller reporting a safety incident do not have the same silence tolerance. Write the local threshold before the run, version it with the prompt and record why the owner accepted it.

Test the audio, not only its transcript

Speech-to-speech systems can produce audio that does not match the text captured beside it. AWS's Nova Sonic test-harness article calls out this divergence directly. Its harness stores final text, audio, tool calls and timing metadata, and it can test full spoken input without a microphone.

For every material entity, compare three values:

{
  "requested": "Tuesday at 3:00 PM",
  "spokenByAgent": "Tuesday at 3:00 PM",
  "recordedByCalendar": "2026-10-27T15:00:00-04:00"
}

The transcript is useful evidence, but it is not authoritative for what the caller heard. The audio is authoritative for the spoken promise. The calendar is authoritative for the booking. Both must agree with the request.

This becomes more important around names, dates, quantities, addresses and confirmation codes. A one-character difference may be a negligible transcript metric and a failed customer task.

Retain the exact audio segment around each material entity. A reviewer should be able to hear it without replaying the entire call. Keep the generated text and timing beside it so engineering can reconstruct the fault.

Treat asynchronous tools as concurrent business intents

Nova 2 Sonic can continue listening and speaking while a tool runs. AWS's asynchronous tool-calling documentation says a pending call is not automatically canceled when the user changes the request. The completed result still reaches the model, which decides how to handle it in the new context.

That makes conversation flow more natural. It also creates a state-management obligation outside the model.

Suppose a caller asks for a flight to Chicago, then changes the destination to Seattle while the first search is running. Two search results can safely coexist. Replace search with a booking or a refund and the consequence changes. The old intent must not commit merely because its tool result arrived first.

Every write-capable call should therefore carry:

{
  "callId": "call-42",
  "intentRevision": 3,
  "effectKey": "return-order-981-rev-3",
  "toolCallId": "tool-77",
  "authorizedAt": "2026-10-20T10:41:00Z"
}

Before committing, the application checks that the intent revision is still current and the action remains authorized. After provider acceptance, it reads the authoritative record back. If the read-back is missing, classify the effect as uncertain and reconcile it. Do not create a fresh effect key and repeat the write.

The broader voice handoff receipt uses the same principle across collaborators: protocol completion, trace evidence, business state and control ownership are different facts.

Gate every run with explicit terminals

The reference gate for this article classifies a run into seven terminals:

• invalid_evaluation: the scenario, contract or end condition cannot be trusted;
• blocked_accuracy: goal state, faithfulness, spoken entities or tool validity failed;
• uncertain_effect: a write may have happened, but no authoritative record was observed;
• divergent_effect: the observed record contradicts the intended effect;
• blocked_experience: the run failed one of the workflow's declared spoken-experience checks;
• unowned_terminal: the disposition is unsupported, control did not return or a human handoff was not acknowledged;
• release_candidate: every required layer satisfied the contract.

The core is deliberately small:

if (!run.simulation.valid || !run.conversation.properEnd) {
  return { status: "invalid_evaluation" };
}

if (!goalStateMatches || !faithful || !speechEntitiesMatch || !toolCallsValid) {
  return { status: "blocked_accuracy" };
}

if (run.businessEffect === "written" && readback === "missing") {
  return { status: "uncertain_effect" };
}

if (!turnTakingPass || !concisePass || !interruptionRecoveryPass) {
  return { status: "blocked_experience" };
}

if (run.terminal === "human_handoff" && !run.handoffAcknowledged) {
  return { status: "unowned_terminal" };
}

I ran the complete dependency-free classifier against 28 synthetic checks. They cover invalid simulations, wrong goal state, spoken-entity divergence, malformed tools, missing and contradictory read-backs, turn and silence failures, unsupported terminals, unacknowledged handoff, safe stop and accepted resolution. All 28 passed.

That proves the classification rule behaves as written. It does not test Nova, a phone network, EVA-Bench, a live integration or a customer outcome.

Build scenarios from failure boundaries

Do not begin with hundreds of happy-path prompts. Begin with the points where the workflow can harm trust or create duplicate work.

For a returns agent, the first scenario set might be:

Then perturb the call: quiet room, background speech, accent, interruption, correction, long tool delay, disconnect and return. Run each against the same initial database state and expected terminal.

AWS's test-harness article notes that Nova Sonic connections time out after about eight minutes and demonstrates reconnecting before the limit with history replay. A production test should add pending effects and control ownership to that replay. Conversation history alone cannot tell the new session whether an external write already committed.

Move from replay to bounded live work

Use a staged release:

1. Offline replay: fixed scenarios, synthetic audio, deterministic tool fixtures and no business writes.
2. Shadow evaluation: real call audio where permitted, but the agent cannot act or speak to the customer.
3. Internal calls: employees use sandbox records and practice every safe stop and handoff.
4. Bounded live pilot: one workflow, one caller cohort, narrow tools, explicit hours and an available human route.
5. Operating review: the workflow owner reviews accepted, blocked, uncertain, divergent and unowned terminals before capacity expands.

Promotion should require a minimum run count chosen in advance, zero unresolved divergent effects, tested handoff acknowledgment and an explicit rollback trigger. The numbers depend on the workflow's risk and traffic. Do not borrow a universal sample size from a vendor post.

The pilot contract should also say who can stop admission. A model change, tool schema change, policy revision, phone-provider change or prompt release can all reopen testing. Version those parts together so the operator can reconstruct what served a specific call.

This follows the larger agent workflow operating contract: answer, action, approval, handoff and refusal are different routes with different evidence. Voice does not merge them. It raises the cost of ambiguity because the customer cannot inspect a long payload before it acts.

Review distributions, not averages

A mean latency or average judge score can hide exactly the calls that should block release. Review the distribution by scenario, persona, noise condition, model version, tool path and terminal.

At the operating review, bring:

• the count of valid and invalid simulations;
• task-accurate calls divided by valid calls;
• spoken-entity mismatches for material fields;
• uncertain and divergent business effects;
• interruption and correction recovery by scenario;
• silence and turn timing against the local contract;
• acknowledged and unacknowledged handoffs;
• the exact model, prompt, tool and policy versions;
• a sample of audio-backed failures, not only transcripts;
• every operator intervention and why it occurred.

The release question is not whether callers liked the voice. It is whether the workflow repeatedly ended in an observed answer, a reconciled action, an acknowledged handoff or a safe stop.

Frequently asked questions