Evals & Observability
What separates demos from deployed — Langfuse and LangSmith, OpenAI tracing, golden datasets, regression testing, and cost telemetry. The instrumentation every production AI system needs.
All Articles
AI Agent Observability: What to Log Before Production
A production logging contract for AI agents: trace IDs, tool calls, retrieval, cost, eval scores, approvals, and alerts before launch.
Trace-to-Eval Builder Build Log
A build log for Trace-to-Eval Builder, a BYOK app that turns agent traces into replayable eval packs.
Agent Runbook Auditor: A BYOK Launch Review Tool for Agent Workflows
A build log for Agent Runbook Auditor, a BYOK OpenAI demo that reviews agent runbooks for launch risk, trace design, eval cases, guardrails, and rollout readiness.
OpenAI Agents SDK Tracing: What It Shows in Production
Use OpenAI Agents SDK tracing as run inspection, not full observability. Configure sensitive data, flushing, trace exports, evals, and approvals.
Langfuse vs LangSmith for Production Observability
Choose Langfuse for self-hosted, framework-neutral traces. Choose LangSmith for managed LangChain evals, review, alerting, and deployment.
One letter, every week. Working systems — not hot takes.
Build logs, agentic engineering decisions, agent failures, evals, and what survives real users. Sent weekly, never more.