Evaluating agent behavior

Unit tests remain genuinely essential for everything mechanical in an agent system — the tool registry, the confirm flow, the rate limiter, the persistence layer (a real system reports "100 tests pass" across exactly these modules) — because that code is deterministic. What unit tests can't meaningfully assert is "does the model produce a good, grounded answer to this kind of question," because the model's actual response is exactly the part that varies. A test suite covering everything except the model's actual behavior is covering the machine around the agent, not the agent itself.

This is closer to an integration test suite than a unit test suite — it hits the actual live SSE endpoint with real questions, and asserts on the shape of correctness (grounded in the right source, or a correct refusal) rather than an exact string match, since exact output text is expected to vary between runs even for a good answer.

The first is the correct behavior for an out-of-scope question — the system working as intended. The second is the dangerous one: a confident-sounding answer with nothing underneath it, which is much harder to notice than an outright error message, precisely because it doesn't look like a failure to a user reading it. A real system logs both patterns into a dedicated table for gap analysis — not because either single instance is necessarily wrong, but because a rising rate of NO_TOOLS_NO_CITATIONS responses over time is exactly the kind of drift that traditional tests, running once and passing, would never catch on their own.

A mocked LLM response can only test "does my code correctly handle a response shaped like X" — it says nothing about whether the actual model, given the actual system prompt and actual tool catalog, produces X (or something good) in the first place. An eval suite's entire value is testing the one part a unit test structurally can't: real model behavior, against real fixtures, hitting the real endpoint.

Both matter, for different reasons: unit tests catch a regression in the deterministic scaffolding around the model (a broken rate limiter, a confirm-step bug) fast and cheaply, on every commit. Evals catch a regression in the model-facing behavior itself (a prompt change that quietly makes the agent worse at knowing when to say "I don't know") — slower to run, more expensive, but checking something unit tests fundamentally cannot.