Evaluating an agent like a chatbot — one prompt, one graded answer — misses the point entirely. Agents produce trajectories: sequences of decisions, tool calls, and recoveries. The interesting failures happen between the first step and the last, and single-turn accuracy is blind to all of them.
Grade the process, not just the answer
An agent that reaches the correct final answer through a broken path is a liability waiting to surface. Robust evaluation asks: did it select the right tools, did it recover from a failed call, did it stop when it should have stopped? A trajectory-level rubric — scoring each step, not just the terminal output — surfaces the brittleness a final-answer check would hide.
Build the harness before the agent
The discipline that makes this tractable is the same one behind good [[Prompt Engineering]]: version everything, hold a fixed evaluation set, and diff behavior across changes. When an agent retrieves context, the retrieval step gets graded too — a bad passage from the [[Retrieval-Augmented Generation]] layer looks like an agent failure but is really a search failure. Separating those is half the battle.
From offline to online
Offline suites catch regressions before deploy; they cannot catch distribution shift. Real user traffic reveals prompts you never imagined. That is why evaluation and [[LLM Observability]] are two ends of the same system — offline scores tell you what changed, production traces tell you what actually happens.
The honest takeaway: an agent is only as trustworthy as the harness that grades it. Invest in the harness first, and the agent quality follows.