AI agent monitoring should start before the first traffic ramp: trace every run, score the outcome, attribute cost, and put risky actions in an approval queue. A normal uptime dashboard can tell you the endpoint returned successfully; it cannot tell you which model step, tool call, retrieval result, or human handoff made the agent unsafe or expensive.
The Verdict: Monitor Agent Runs, Not API Requests
The production unit is the agent run, not the chat endpoint. If the only thing you monitor is request latency, error rate, and infrastructure saturation, you will miss the failures that make agents expensive: repeated model calls, stale retrieval, a tool returning plausible garbage, an approval that never fires, or a prompt version that quietly changed the agent's behavior.
Gartner put a hard business frame around the risk: it predicted that over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value, or inadequate risk controls. Treat that as an operating warning. Monitoring is not a dashboard you add after launch. It is the control layer that proves the agent is useful enough, cheap enough, and bounded enough to receive more traffic.
The run record should be the first object you design:
The Minimum Production Signal Set
The minimum monitoring set is run health, tool reliability, model cost, retrieval quality, safety events, approval state, and outcome quality. That set is small enough to ship before launch and broad enough to explain why an agent failed after launch.
IBM frames AI agent observability around metrics, events, logs, and traces, with agent-specific signals such as token usage, tool interactions, decision paths, LLM calls, failed tool calls, escalations, alerts, and end-to-end traces. That is the right shape, but the production line is stricter: every signal needs an owner and an action.
For example, a tool failure counter is not enough. A useful tool signal records:
{
"run_id": "run_...",
"trace_id": "trace_...",
"span_type": "tool_call",
"tool_name": "search_docs",
"tool_version": "2026-06-05",
"status": "failed",
"error_class": "timeout",
"input_hash": "sha256:...",
"output_hash": null,
"retryable": true,
"owner": "knowledge-platform"
}That record lets the on-call engineer decide whether to roll back an agent prompt, fix the retrieval service, rate limit a tenant, or disable a risky action. Monitoring that does not lead to a decision is just storage.
Trace the Agent Path, Then Add Evaluations
Tracing explains what happened; evaluations decide whether the run was good enough. You need both before traffic grows.
The OpenAI Agents SDK tracing docs are a useful baseline for agent teams because the SDK records LLM generations, tool calls, handoffs, guardrails, and custom events during an agent run. It traces Runner.run, Runner.run_sync, and Runner.run_streamed by default, wraps agent execution, generation calls, function calls, guardrails, and handoffs, and lets teams add custom processors for other destinations.
That native trace is strong for teams building on the OpenAI Agents SDK. The moment your system crosses frameworks, providers, workers, queues, or MCP servers, keep the trace IDs compatible with a vendor-neutral shape. OpenTelemetry GenAI semantic conventions are marked stable and define conventions for events, exceptions, metrics, model spans, agent spans, and technology-specific systems including OpenAI, Anthropic, Azure AI Inference, AWS Bedrock, and MCP.
The practical stack decision is:
Langfuse is useful when the team wants an open-source LLM engineering layer with traces, model parameters, token usage, prompt management, datasets, experiments, evaluation scores, custom dashboards, and self-hosting. Langfuse tracing captures LLM calls, retrieval steps, tool executions, custom logic, timing, inputs, outputs, and metadata.
LangSmith is useful when the team already works in the LangChain or LangGraph ecosystem and wants observability tied to evaluation and feedback workflows. LangSmith Observability exposes individual traces and production-wide performance metrics, with trace filtering, export, sharing, comparison, rules, webhooks, online evaluations, user feedback queues, and inline annotation.
Datadog is useful when the production question is not just "what did the model do?" but "what broke in the whole application path?" Datadog LLM Observability monitors performance, costs, traces, token usage, and errors; it represents application requests as traces and groups LLM calls with contextual operations such as tool calls or preprocessing steps. It also evaluates quality, privacy, and safety, scans and redacts sensitive data, identifies prompt injections, and surfaces anomalies in duration and error rate.
If the decision is between the agent framework and the observability surface, start with the framework choice first. The monitoring path differs between OpenAI Agents SDK and LangGraph. If the question is specifically the LLM observability product, compare the operating model in Langfuse vs LangSmith.
Create the run envelope
Create one run record before the first model call. Attach the user, tenant, workflow, agent version, prompt version, model route, feature flag, and parent request trace ID.
Instrument spans for every agent decision
Record spans for model calls, retrieval, tool calls, guardrails, handoffs, queue waits, approvals, and final output assembly. A span without a parent trace is a debugging dead end.
Attach cost and version metadata
Store token counts, provider response IDs, model names, cache state, prompt version, tool version, and index version. Cost spikes and quality regressions are usually versioned events.
Score the run
Attach at least one evaluator result to each meaningful workflow: task completed, cited source present, unsafe output blocked, tool result used correctly, human escalation needed, or user feedback captured.
Route risky actions to approval
When the agent can send, write, delete, merge, purchase, refund, or message a customer, monitoring must include the approval decision, not just the proposed action.
Cost Monitoring Needs Attribution, Not Token Totals
Cost monitoring is only useful when it names the owner of the spend. Model totals tell you the provider bill; attribution tells you which workflow, customer tier, prompt version, retrieval path, or tool loop created it.
Trace volume itself can also become a cost line. Langfuse Cloud pricing lists a free Hobby plan with 50k units per month included, Core at $29 per month with 100k units included and additional usage at $8 per 100k units, and Pro at $199 per month with 100k units included and the same additional unit rate. LangSmith pricing lists Developer at $0 per seat per month with up to 5k base traces per month and Plus at $39 per seat per month with up to 10k base traces per month; base traces have 14-day retention at $2.50 per 1k traces after included usage, while extended traces have 400-day retention at $5.00 per 1k traces.
Those prices do not mean "pick the cheapest trace store." They mean the monitoring plan needs a retention policy. Keep complete traces for launch, high-risk workflows, incidents, and labeled eval samples. For routine low-risk flows, keep the run record, aggregate metrics, redacted inputs, version metadata, and enough spans to diagnose regressions.
The cost panel should answer these questions without a notebook:
- Which workflow created the most model spend this week?
- Which prompt version increased tokens per successful run?
- Which retrieval path caused repeated tool calls?
- Which tenant or user tier is unprofitable under current limits?
- Which agent should move to a cheaper model route without hurting eval scores?
- Which traces must be retained because they include approvals, incidents, or human labels?
The important ratio is not cost per token. It is cost per successful outcome. A run that costs more and completes the task cleanly can be cheaper than a low-cost run that retries, escalates, and creates support work.
A Practical Dashboard Layout
A useful AI ops dashboard has a small set of core surfaces: launch gates, run explorer, trace detail, cost attribution, and approval queue. Each answers a different operating question, and each should share the same run IDs.
create table ai_agent_runs (
run_id text primary key,
trace_id text not null,
tenant_id text,
user_tier text,
workflow_name text not null,
agent_name text not null,
agent_version text not null,
prompt_version text not null,
model_name text not null,
started_at timestamptz not null,
finished_at timestamptz,
final_state text not null,
failure_class text,
input_tokens integer,
output_tokens integer,
cached_tokens integer,
cost_usd numeric,
approval_status text,
eval_status text,
eval_score numeric,
created_at timestamptz not null default now()
);Build the dashboard from that table, then join trace spans, tool calls, retrieval events, approval decisions, and evaluator results.
The launch gate is the executive surface for engineering: it says whether the agent earns more traffic. The run explorer is the support and on-call surface: it says what happened. The trace detail is the engineering surface: it says why. The approval queue is the control surface: it says what the agent is allowed to do.
Do not hide approvals in Slack messages or email notifications. Store the proposed action, risk tier, full context, approver, decision, and trace link. A rejected approval is a product signal. A timed-out approval is an operations signal. An approved action with a later bad outcome is an eval sample.
What Breaks First in Production
The first production break is usually missing context, not model intelligence. The model may be fine, but the team cannot answer which prompt version ran, which tool returned the bad value, which retrieval index produced the citation, or who approved the action.
Watch these failure modes before launch:
- Unversioned prompts: if prompt edits are not stored with run records, every regression turns into guesswork.
- Tool status that only says success: a tool can return stale, partial, or unauthorized data while still returning a successful response.
- Trace sampling that drops rare failures: low-volume failures are often the ones that matter most: refunds, account changes, code edits, customer messages, or admin actions.
- Sensitive data in traces: OpenAI notes that
generation_span()stores LLM inputs and outputs,function_span()stores function inputs and outputs, andtrace_include_sensitive_dataisTrueby default unless disabled. Decide what can be captured before production traffic arrives. - Cost dashboards with no owner dimension: a model spend chart without tenant, workflow, feature, and outcome fields cannot drive action.
- Approvals outside the trace: if the approval decision is not attached to the run, you cannot audit the action or improve the risk policy.
- Evals disconnected from incidents: every incident should add at least one regression case to the eval set.
The fix is a simple rule: every production agent action needs a run record, every run needs a trace, every trace needs cost and version metadata, every risky action needs an approval state, and every meaningful outcome needs an eval signal.
What is AI agent monitoring?
AI agent monitoring is the production practice of tracking the full agent run: model calls, tool calls, retrieval, state changes, handoffs, cost, approvals, safety events, and final outcome. It is broader than uptime monitoring because a completed request can still be wrong, unsafe, or too expensive.
How is agent monitoring different from LLM monitoring?
LLM monitoring tracks an individual model call. Agent monitoring tracks the workflow that surrounds it: planning, retrieval, tool execution, memory, guardrails, handoffs, approvals, and whether the task was completed.
What metrics should production AI agents track first?
Start with run success, failure class, tool failure rate, tail latency, token usage, cost by workflow, eval result, unsafe-output events, prompt version, retrieval index version, and approval state. Add product-specific outcome metrics once the basic run record is reliable.
Do we need OpenTelemetry for AI agent monitoring?
Use OpenTelemetry-compatible trace IDs and GenAI span conventions when the system crosses providers, frameworks, services, queues, or MCP servers. A single-framework prototype can start with native tracing, but a production system needs trace continuity across the whole request path.
Should every agent action require human approval?
No. Approval belongs on actions with irreversible, external, financial, customer-visible, or security-sensitive effects. Low-risk read-only actions can run automatically, but their traces and outcomes still need to be logged.
Scope Your AI Ops Dashboard
Build the run, cost, failure, and approval layer your production agents need before traffic scales.
