Agent Evaluation Frameworks: How to Test AI Agents in Production

An agent evaluation framework grades the whole trajectory, every tool call, state change, and intermediate step, not just the final answer, because in an agent one early mistake compounds into ten. The framework you pick matters far less than getting that grading layer in before your users become the test suite.

Here is the short version. If your team already lives in LangChain, start with LangSmith. If you want tests that run in CI like any other test suite and you are comfortable in Python, start with DeepEval. If you are an AI-native product team that wants logging and evals in one hosted place, Braintrust. If you need to self-host for data-control reasons, Arize Phoenix. If the thing you are grading is a retrieval pipeline, add Ragas on top of whichever you pick. None of these decisions is the important one. The important one is what you measure, and almost every team gets that wrong first.

Why grading the final answer is not enough

A chatbot eval checks one thing: prompt in, response out, does the response match. An agent breaks that model. Agents operate over many turns, calling tools, modifying state, and adapting to intermediate results, and those same properties that make them useful make mistakes propagate and compound. An agent that books the wrong flight can produce a perfectly polite confirmation message. Grade the message and it passes. Grade what happened and it fails.

So agent evals split the thing you measure into two layers, and you need both:

• The transcript (also called the trace or trajectory) is the complete record of a run: every output, tool call, reasoning step, and intermediate result. For an API-based agent it is the full messages array at the end of the run.
• The outcome is the final state in the environment. A flight-booking agent might say "your flight has been booked," but the outcome is whether a reservation actually exists in the database. The transcript tells you how; the outcome tells you what.

One consequence engineers miss: when you evaluate "an agent," you are evaluating the harness and the model together. The scaffold that processes inputs, orchestrates tool calls, and returns results is part of what you are grading. Swap the model and keep the harness, or refactor the harness and keep the model, and your numbers move. Version both in your eval metadata or you will chase ghosts.

The three graders, and when each one earns its cost

Every agent eval is built from three kinds of grader. The design work is choosing the right one for each thing you care about, not picking a favorite.

The rule that follows: verify with code wherever a fact is checkable, judge with a model where quality is subjective, and use humans to calibrate the judge, not to grade every run. A coding-agent eval leans on deterministic unit tests for correctness and adds a model-based rubric for code quality only where tests cannot reach. A support-agent eval checks the end state (was the ticket resolved), constrains the transcript (did it finish in under ten turns), and uses a rubric for tone. Same three graders, different mix per task.

This is why the benchmarks that hold up are the ones with verifiable outcomes. SWE-bench Verified grades coding agents by running the repository's test suite: a fix counts only if it makes the failing tests pass without breaking the passing ones, and scores on it climbed from 40% to over 80% in a single year precisely because the grader is not arguable. tau2-bench does the multi-turn version, simulating retail and airline support with one model playing the user while the agent works. Borrow the pattern: pin your evals to state you can check, not vibes you have to trust.

The frameworks, compared

Here is the live landscape for the tools production teams actually reach for, with this-week pricing.

DeepEval treats evals like unit tests. You pip install -U deepeval, write a test case, pick a metric, and run deepeval test run in the same CI job as the rest of your suite. Its GEval metric is a research-backed LLM-as-a-judge you point at any custom criterion, scores land on a 0 to 1 scale, and a threshold (0.5 by default) decides pass or fail. For agents specifically, the recommended path is component-level evals through tracing rather than black-box grading of the final output, so you can see which step in the trajectory regressed. Confident AI is the hosted layer on top: free for up to two seats, then $200/month for Starter (unlimited seats, $1 per GB-month, 5k online eval runs) and $2,000/month for Team (50k online eval runs). Start local and free; add the cloud when you need shared dashboards.

LangSmith is the default if your agent is built on LangChain or LangGraph, because the traces are already flowing. It combines tracing, datasets, and evaluators in one hosted product. The Developer tier is $0 per seat with up to 5k base traces a month and a single seat; Plus is $39 per seat per month with 10k base traces and unlimited seats; Enterprise is custom. The lock-in cuts both ways: seamless if you are in the ecosystem, a data-egress question if you are not.

Braintrust is built for teams whose whole product is an LLM app and who want logging, datasets, playgrounds, and evals in one hosted surface. The Free tier is $0 with $10 of credits, 1 GB of processed data, and unlimited users; Pro is $249/month with $249 in credits and 5 GB, with 6 to 12 months free for qualifying startups. It is the least framework-opinionated of the hosted options, which is the point if your stack is heterogeneous.

Arize Phoenix is the answer when self-hosting is a requirement, not a preference. It is open source, Apache 2 at its core, and built on OpenTelemetry, so it slots into an existing observability pipeline instead of replacing it. You send traces, score outputs to catch failures and regressions, iterate on prompts against real production examples, and run experiments that compare changes on the same inputs. No per-seat bill, and your trace data never leaves your infrastructure.

Ragas is not a competitor to the others; it is what you add when the thing under test is retrieval. It is an open-source metrics library for LLM apps with custom metrics via decorators, built-in dataset management, and integrations with LangChain and LlamaIndex. Wire its retrieval and answer metrics into DeepEval, LangSmith, or Phoenix and grade the RAG layer with purpose-built scores rather than generic ones. If your agent's failures trace back to bad chunks, this is where the signal is, with retrieval-specific scores that generic answer-quality metrics miss.

How to stand up agent evals without boiling the ocean

The teams who get this right start small and start from real failures, not from a coverage target. Two useful reference points: Descript built evals around three plain dimensions (do not break things, do what I asked, do it well) and evolved from manual grading to LLM graders with periodic human calibration, running separate suites for quality benchmarking and regression. Bolt started evals only after they already had a widely used agent, then built a full system in three months that grades with static analysis, browser agents, and LLM judges. Late is common. Never is the failure.

1. 1

   Mine your traces for the first ten cases

   Do not invent test cases. Pull ten real transcripts where the agent did something wrong or something notably right. Each becomes a test case: the input, the tools available, and the outcome you expected. This is also why a tracing layer comes first. If you cannot see the trajectory, you cannot grade it, and you cannot build the dataset. The what-to-log checklist for production agents covers the minimum you need captured.

2. 2

   Assign a grader per assertion

   For each case, split what you care about into checkable facts and subjective quality. Facts get code graders: did it call the right tool with the right parameters, did the end state change correctly, did it finish under the turn limit. Quality gets a rubric-based LLM judge. Do not reach for a model where a string match will do.

3. 3

   Calibrate the judge before you trust it

   Run your LLM judge against a handful of cases a human has already scored and check that it agrees. An uncalibrated judge is a random number generator with good grammar. Recalibrate whenever you change the judge model or the rubric.

4. 4

   Wire it into CI as a gate, not a report

   Put the suite in the same pipeline as your tests so a regression blocks the merge. Set thresholds you will actually enforce. An eval that emails a dashboard nobody reads is theater; an eval that fails the build is a control.

What breaks once real traffic hits

Evals that looked healthy in development fail in production in predictable ways, and the good frameworks are the ones that make these visible.

• LLM judges drift and cost real money. Every model-based grade is an API call; at production volume the judge bill is a line item, and a model update silently shifts your scores. Sample rather than grade every event, pin the judge model version, and alert on judge-score drift the same way you alert on latency.
• Deterministic graders rot. A string-match assertion that was correct in March breaks when a valid new phrasing appears, and it fails closed, flagging good behavior as bad. Prefer outcome and state checks over output-text matches wherever the environment lets you verify the real end state.
• The harness changes underneath you. A prompt tweak or a new tool is a change to the thing being evaluated. Without harness and model versions in your eval metadata, a score move is unattributable. Log both on every run.
• Coverage hides in the tail. Ten happy-path cases pass forever while the failures live in the long tail you never wrote a case for. This is why the dataset has to grow from production incidents, not from a one-time authoring sprint.

None of this is a reason to skip evals. It is the reason the grading layer has to be observable, versioned, and enforced, which is exactly the difference between an agent that survives real users and a demo that impressed everyone once. For the judge design itself, we go deeper in building an LLM-as-a-judge you can trust.

Agent evaluation framework vs observability tool, what's the difference?

Observability shows you what your agent did in production (traces, logs, latency, cost). An evaluation framework grades whether what it did was correct, against a dataset with known-good outcomes. You need both, and they overlap: Phoenix, LangSmith, and Braintrust bundle tracing and evals, while DeepEval focuses on the grading and reads traces you already capture.

Do I need a framework, or can I write my own evals?

You can hand-roll assertions for a handful of cases, and for a first pass that is fine. A framework earns its place once you need shared datasets, LLM-judge calibration, CI integration, and dashboards the whole team reads. Most teams outgrow home-rolled scripts within a quarter of shipping.

How many test cases do I need before launch?

Start with ten drawn from real transcripts, not a round number picked for comfort. Cover your highest-risk paths first, then grow the set from every production surprise. A small, real, growing dataset beats a large synthetic one that never changes.

Open source or hosted?

Start open source (DeepEval, Phoenix, Ragas) to learn what you actually need and to keep trace data in house. Move to a hosted layer (Confident AI, LangSmith, Braintrust) when shared dashboards, collaboration, and managed scale matter more than the per-seat or per-trace bill. The dataset moves with you either way.

Start the Rescue

Agents already in production but failing in ways you cannot see? We instrument the traces, build the eval harness, and turn your failures into a suite that gates every deploy.