8 min read

Agent Evals

Your 2,500 tests can’t see the one thing your customers actually touch: how the AI behaves. Unit and API tests mock the model, so they prove your plumbing works while a prompt tweak silently breaks tool selection, a model swap starts leaking refusals, or a “helpful” description change makes the assistant stop calling tools. The eval suite is the regression net for that layer: real model, real tools, real database, in CI.

It earned its keep before it ever shipped. Its first honest run caught a live bug (a tool description that made the model ask for confirmation in prose instead of calling the tool), and its review round caught a second (a self-contradictory test transcript the model was correctly refusing to play along with).

This guide is the concepts + workflow tour. The operational reference (case schema, every env var, cost table, CI contract) ships in your copy of the kit at src/lib/ai/evals/README.md.

Why Regular Tests Aren’t Enough

Mocked tests encode the assumption under test: a mocked model always “behaves.” The failure mode that motivated this suite: a memory-consolidation feature silently dropped new facts because a conservative cheap model returned a perfectly valid [], so every mocked test stayed green. Only running the real model caught it.

Evals flip the contract: the model is real, and nondeterminism is handled by what you assert, not by mocking it away.

Assert outcomes, not wording. The phrasing of “Sure, I’ve invited her!” varies per run; these don’t: the right tool was proposed, with the right args, the approval pause happened, the DB row exists afterward.
Set-semantic tool checks. Required tools present, load-bearing args pinned. Order and extra calls are tolerated unless a case opts into strictness, because exact-sequence matching punishes valid behavior.
Binary LLM judge only where code can’t check (e.g. “is this answer grounded in the tool result?”): pass/fail plus a reason, never 1–5 scores, and structurally excluded from anything that gates CI.
pass^k trials. Consistency-critical cases run 3 times and must pass all 3. “Passed once” and “passes reliably” are different claims.

Tech Stack, and Why It’s Zero Dependencies

The whole suite runs on the Vitest 4 you already have. Nothing new in package.json.

Considered	Verdict	Why
Plain Vitest harness	Built this	~300 lines you own and can read; no third party to stall or break under you
evalite	No	Stalled single-maintainer beta; its stable line doesn’t even support Vitest 4
autoevals (Braintrust)	No	Healthy, but hardwires an OpenAI client that ignores your configured provider
promptfoo	Docs recipe only	Red-team gold standard, but a second YAML-based test universe (and OpenAI-owned)
Eval platforms	Graduate later	Langfuse / Braintrust / LangSmith via AI SDK telemetry when you outgrow in-repo evals

The judge and every eval call ride the kit’s own provider factory: whatever LLM_PROVIDER your .env.local says, evals use.

Architecture

datasets/*.jsonl ──▶ runner ──▶ streamChat()  (the REAL production seam:
   (golden cases,     │          real system prompt, role-filtered tools,
    Zod-validated)    │          real model, real Postgres)
                      ▼
              trajectory normalizer   ◀── the ONLY file touching SDK step shapes
                      │                   (= the one-file AI SDK v7 migration seam)
                      ▼
              scorers + assertions    deterministic first; binary judge where needed
                      ▼
              reporter                summary table, results.json, baseline delta

Everything lives in src/lib/ai/evals/: three suites (core-agent, security, memory), the harness, and a committed baseline.json that works like a snapshot. Diffs get reviewed in PRs; it updates only deliberately.

Design decisions worth knowing:

Two tiers. regression cases gate PRs and are deterministic-only (the schema rejects a judge scorer on a regression case, so flaky judgment can never block a merge). capability cases report without blocking.
HITL tools never execute. For approval-gated tools the assertion is the approval pause; no Clerk mutations, ever. See Write Actions & Human-in-the-Loop.
Infra errors ≠ failures. A provider outage marks cases error, loudly and distinctly, never recorded as a behavioral regression.
Your usage dashboard stays honest. Evals bypass usage recording entirely (proven: ai_usage row count identical before and after a run) and print their own cost instead. See AI Usage Dashboard.
The security suite is part of the product. Cross-tenant probes, member→admin escalation, prompt injection via RAG chunks and stored memories, HITL bypass, system-prompt extraction: your multi-tenancy promises are tested, not asserted. The cross-tenant case is mutation-verified — deleting the organizationId scope from retrieval makes it fail deterministically.

Running the Suite

Prereqs: Postgres up, your normal .env.local (LLM_PROVIDER + LLM_API_KEY; memory and RAG cases also want embeddings config, and self-skip with a notice without it). Eval runs cost real API credits — measured numbers below.

# Cheapest first touch — one suite (~$0.02, ~30s)
npx vitest run --config vitest.config.eval.ts src/lib/ai/evals/memory.eval.test.ts

# Full suite, all tiers
make eval

# Exactly what the PR gate runs
EVAL_TIER=regression npm run eval

# Try a different model (must be in your provider's curated list)
EVAL_MODEL=google/gemini-3.5-flash npm run eval

# Accept current behavior as the new baseline (refuses under a tier filter)
npm run eval:baseline

Measured on the kit’s own repo (OpenRouter): full suite ≈ $0.56 / 3 min on claude-sonnet-4-6; regression tier ≈ $0.26 / 2 min on gemini-3.5-flash.

In CI, evals.yml runs the regression tier on PRs touching AI paths and the full suite nightly. No LLM_API_KEY secret → the job skips green with a visible notice, so fresh forks never start red. An optional EVAL_MODEL repository variable pins PR runs to a cheap model.

Trusting It

A test suite you’ve never seen fail is a decoration. Two quick proofs:

# Misconfiguration fails fast — ~1 second, zero tokens, lists valid models
EVAL_MODEL=nope npm run eval

And for a behavioral failure: edit any case in datasets/core-agent.jsonl to expect the wrong tool. The run goes red with the scorer explaining exactly which tool was proposed instead. Revert, and you’ve watched the whole loop work.

Extending It (with AI, of Course)

The core workflow is one sentence: a production failure becomes a JSONL line, and that failure can never come back.

Each case is one line (input, seed data, expectations):

{"id": "tool-select-team-list", "suite": "core-agent", "tier": "regression", "role": "owner", "input": "How many people are on my team?", "expect": {"tools": {"required": [{"name": "getTeamMembers"}]}}}

Let the AI do it; it knows this suite through the scoped rules:

“The assistant invited the wrong person when two members shared a first name — use the fix-bug skill, and add an eval case reproducing it.”
“Use the new-feature skill to add an archiveProject tool” — the testing rule now requires eval cases for LLM-judgment features, so the AI adds them as part of the feature, not as an afterthought.
“Compare gemini-3.5-flash against the baseline on the regression tier and tell me if it’s safe to switch.”

Two habits keep the suite honest: write both directions for every behavior (should-call and should-not-call; one-sided evals create one-sided optimization), and if you’re passing 100% forever, your cases are too easy. Promote hard ones from capability once they stabilize.

When you outgrow terminal output (trace exploration, dashboards, production sampling), the graduation path is the AI SDK’s telemetry into Langfuse (self-host) or Braintrust/LangSmith (managed). The harness stays; the platform is additive.

AI Development Framework — The scoped rules and skills the eval workflow rides on
Prompt Library & Skills — fix-bug, new-feature, and the full SDLC flow
Write Actions & Human-in-the-Loop — The approval-gated tools the suite asserts against
Knowledge Base (RAG) — The retrieval pipeline the security suite injects against
AI Usage Dashboard — The cost tracking evals deliberately bypass

Ready to build with VibeReady?

Get the full AI-native SaaS foundation with production infrastructure, AI development framework, and all integrations.

Get VibeReady — From $149