Skip to content
All writing
LLM Evaluation#Agents#LLM#MLOps#Research

Evaluation of AI Agents

Single-turn accuracy tells you almost nothing about an agent. How to evaluate trajectories, tool use, and failure recovery.

March 11, 20262 min read

Evaluating an agent like a chatbot — one prompt, one graded answer — misses the point. Agents take trajectories. The interesting failures happen between the first step and the last.

What single-turn eval misses

An agent that reaches the right final answer through a broken path is a liability waiting to surface. Real evaluation has to grade the process:

  • Did it pick the right tools?
  • Did it recover from a failed tool call?
  • Did it stop, or loop forever?
  • Did it stay grounded in retrieved evidence?

An agent can be right for the wrong reasons and wrong for the right reasons. Grading only the final answer hides both.

Four axes I actually measure

@dataclass
class AgentEval:
    task_success: float        # did it achieve the goal?
    tool_precision: float      # right tools, right arguments
    grounding: float           # claims supported by evidence
    efficiency: float          # steps vs. optimal path

1. Task success

Outcome-based: did the final state satisfy the goal? This is necessary but, on its own, dangerously incomplete.

2. Tool-use precision

Did the agent call the correct tool with valid arguments? Wrong-tool selection is one of the most common silent agent failures.

3. Grounding

For any factual claim, is there supporting evidence in what the agent retrieved? Ungrounded confidence is the hallucination failure mode in disguise.

4. Efficiency

Compare the trajectory length to an optimal path. A ballooning step count is an early warning of reasoning drift and runaway cost.

Build a regression harness, not a leaderboard

The goal isn't a single score — it's catching regressions before deploy. Curate a suite of representative trajectories with known-good behavior, run it in CI, and gate releases on it.

Treat evals like tests. A change that improves your headline metric but breaks three trajectory checks is a regression, not an improvement.

LLM-as-judge, carefully

Model-graded evaluation scales, but judges have biases (length, position, self-preference). Calibrate the judge against human labels on a holdout, and keep a human in the loop for high-stakes categories.

The bottom line

You cannot trust an agent you cannot evaluate along its whole trajectory. Success rate is the headline; tool precision, grounding, and efficiency are the story.

Share
← Back to writing

Related reading