Evaluating agent behavior

A unit test can assert that a function returns the right value for a given input, every time, deterministically. An LLM's response to the same prompt isn't guaranteed to be identical twice. Evaluating an agent needs a different kind of test.

Advanced

3 min read

Why ordinary unit tests aren't enough on their own

Unit tests remain genuinely essential for everything mechanical in an agent system — the tool registry, the confirm flow, the rate limiter, the persistence layer (a real system reports "100 tests pass" across exactly these modules) — because that code is deterministic. What unit tests can't meaningfully assert is "does the model produce a good, grounded answer to this kind of question," because the model's actual response is exactly the part that varies. A test suite covering everything except the model's actual behavior is covering the machine around the agent, not the agent itself.

What an eval harness actually is

fixtures.ts:
  [
    { question: "What's the visitor parking policy?",
      expectedSource: "visitor-parking-policy.md" },
    { question: "How do I reset my portal password?",
      expectedSource: "password-reset-guide.md" },
    { question: "What's the capital of France?",
      expectedFallback: true },   // should say "I don't know" / decline
    ... (dozens more)
  ]

run-eval.ts:
  for each fixture:
    hit the REAL, live agent endpoint (not a mock)
    check: did the response cite the expected source?
      OR, for out-of-scope questions: did it correctly decline
      rather than confidently making something up?
    record pass/fail

This is closer to an integration test suite than a unit test suite — it hits the actual live SSE endpoint with real questions, and asserts on the shape of correctness (grounded in the right source, or a correct refusal) rather than an exact string match, since exact output text is expected to vary between runs even for a good answer.

The two failure modes worth naming and detecting separately

NO_INFO_FALLBACK       - the agent correctly recognized it doesn't
                           have grounding for this and said so

NO_TOOLS_NO_CITATIONS  - the agent answered confidently, but with
                           nothing backing the answer up — no tool
                           was called, no source was cited

The first is the correct behavior for an out-of-scope question — the system working as intended. The second is the dangerous one: a confident-sounding answer with nothing underneath it, which is much harder to notice than an outright error message, precisely because it doesn't look like a failure to a user reading it. A real system logs both patterns into a dedicated table for gap analysis — not because either single instance is necessarily wrong, but because a rising rate of NO_TOOLS_NO_CITATIONS responses over time is exactly the kind of drift that traditional tests, running once and passing, would never catch on their own.

Why this needs to run against the real thing, not a mocked model response

A mocked LLM response can only test "does my code correctly handle a response shaped like X" — it says nothing about whether the actual model, given the actual system prompt and actual tool catalog, produces X (or something good) in the first place. An eval suite's entire value is testing the one part a unit test structurally can't: real model behavior, against real fixtures, hitting the real endpoint.

Evals are a complement to unit tests, not a replacement

Both matter, for different reasons: unit tests catch a regression in the deterministic scaffolding around the model (a broken rate limiter, a confirm-step bug) fast and cheaply, on every commit. Evals catch a regression in the model-facing behavior itself (a prompt change that quietly makes the agent worse at knowing when to say "I don't know") — slower to run, more expensive, but checking something unit tests fundamentally cannot.