FAFO: Learn to Write Evals
Write graders, catch bad agent behavior, and check whether a change helped. Read this long AF post, or make your coding agent teach you with the included learn-evals skill.
Maybe you’re trying to build a software factory, maybe an internal support agent. Maybe you just keep hearing about evals. This post walks you through what they are and how to write them.
But this blog post is pretty freaking long. I’d probably just point my agent at it and ask the agent to teach me. Hell, that’s actually what I did when I learned about evals. Fed the robot a bunch of links/articles, then had it walk me through stuff.
So I made a skill you can use instead of reading this long af post. This way, your coding agent can help you FAFO to learn how evals work.
Ok, so about that skill…
Want your coding agent to walk you through it?
If you wanna skip the post and have your coding agent walk you through the same lessons, grab the repo. The teaching skill comes with it:
git clone https://github.com/pandemicsyn/fafo.git
cd fafoOpen your coding agent in that directory and ask:
Use the learn-evals skill to teach me evals. Start with lesson 1. Let me predict the result and try the exercise before showing me the answer.
You can also pick learn-evals in your client’s skill picker, or use /learn-evals where that works. Ask for a hint. Skip something you already know. Come back later. The agent can help install dependencies when you’re ready to run code. If it can’t find the skill, ask it to read .agents/skills/learn-evals/SKILL.md.
Want the tutor available in another project? You can still install it with npx skills add pandemicsyn/fafo --skill learn-evals. The official vitest-evals skill covers the library-specific bits; add it with npx skills add getsentry/vitest-evals when you get to writing evals.
Rather read and work through it yourself? Ok, grab a snack and buckle up. This is probably like a 20-minute read.
The demo/project repo
The fafo repo is built with TypeScript, Flue, and vitest-evals, although the stack here really doesn’t matter. What you learn here will apply to any language/framework probably. I dunno, I’m not an expert.
This repo has a few deliberately broken exercises. We’ll fix the checks, find out what they still miss, and compare changes without falling into some eval traps.
Being comfortable with TypeScript and tests helps, but you can always use the agent skill above. No eval experience needed.
The setup
It’s an issue-triage agent. Give it a bug report. It searches a fictional tracker, asks for missing facts, and either creates an issue or adds a comment to an existing duplicate.
Here are the rules for our pretend setup:
- A complete issue report includes the affected feature, reproduction steps, expected behavior, and observed behavior. Ask for missing facts before writing.
- An open issue is a duplicate when the feature, trigger, and observed failure match. Similar titles alone aren’t enough. Under this application’s policy, a closed issue doesn’t count as an open duplicate.
- Search before writing and read candidate details before deciding they’re duplicates. If search fails, explain the limitation and don’t write.
- Add exactly one comment to the matching duplicate, or create exactly one new issue. Preserve the user’s facts and confirm only work that succeeded.
The agent has access to four tools: search_issues, get_issue, create_issue, and add_comment. The tracker runs locally; there is no GitHub account to connect or deployment step. Flue runs the agent, and vitest-evals gives us a test structure for running it and grading the evidence.
Your coding agent of choice can absolutely help with the exercises if you don’t wanna write code. The issue-triage agent is the thing we’re testing. Later, a separate model will judge one property of its output.
Following along yourself? Clone the repo using the commands above if you haven’t already. From the fafo directory, install its dependencies with Node 22.19 or newer:
npm ciThe first six lessons use synthetic examples (they’re fake): inputs and outputs constructed so you can reproduce the same failures without an API key. Lessons 3, 5, 6, and 9 have broken functions in exercises/. Those lesson commands fail on purpose. Fix the function. Leave the expected answers alone. Ordinary npm test checks the repo itself and should pass throughout.
Ok, let’s fuck around and see what we can find out…
1. First look
From the fafo directory, run:
npm run examplesThis prints a prepared example as JSON in your terminal. It doesn’t run a model or create a real issue. Your job in this lesson is to inspect the evidence; there’s no code to fix yet.
The example’s input is this bug report:
In the issue list, type a word into search and press Escape. Expected: search clears and all issues return. Actual: the input clears, but the list stays filtered until I refresh.
What you expected: one new issue about issue search, with the user’s reproduction steps.
Now look at three fields in the printed JSON. Here’s the relevant part, with other fields omitted:
{
"output": {
"reply": "Created ISS-1 with your reproduction steps.",
"before": { "issues": [] },
"after": { "issues": [] }
}
}output.reply is what the agent said; before.issues and after.issues are the stored issues on either side of the task. We’d expect to have a new issue in after.issues.
What you got: a confident sentence, and a tracker that didn’t change. A successful write would have put an issue object in after.issues.
Now run the second example:
npm run examples -- lying-toolThis one adds a tool receipt. Look at trace[0].result: it contains an issue object with id: "ISS-1", as though create_issue succeeded. Then look at output.after.issues again. Still empty. There should be an issue listed here. The receipt claims a write that the tracker doesn’t contain.
Try it: For each example, point to the claim of success and the field that contradicts it. Explain why you would check the tracker’s stored issues before telling the user their bug was filed.
A few names for the pieces you just saw. We’ll use them more below:
| Term | In this example |
|---|---|
| Case or task | The report, initial tracker state, and expected behavior |
| Trial | One execution of that case |
| Harness | Code that prepares the state, runs the application, and captures evidence |
| Transcript or trace | Messages, tool calls, and tool results from the interaction |
| Grader | A check of a particular requirement |
| Outcome | The resulting tracker state |
2. Inspect failures before choosing your checks
No issue got written. Fine, we can check for that. But what if an issue exists and it’s still wrong?
npm run examples -- --list
npm run examples -- wrong-feature
npm run examples -- double-writeIn wrong-feature, inspect output.after.issues[0].feature: it says notifications, though the input describes issue search. In double-write, count the objects in output.after.issues: there are two for one report.
Both examples contain an issue and still get the job wrong. A grader that only asks whether any issue was created would accept both. This lesson’s output is notes, not code.
For each failure, write three things: what happened, why it matters to the user, and the evidence that would catch it.
| Observation | User impact | Candidate check |
|---|---|---|
| Search bug filed under notifications | Report reaches the wrong area | New issue’s feature matches the report |
| Two issues created for one report | Duplicate work for maintainers | Exactly one new issue |
| Reproduction trigger omitted | Maintainer cannot reproduce the bug | Compare the report’s facts with the stored issue |
That’s error analysis. Look at the failures, then decide what to measure. A generic “helpfulness: 1–5” score doesn’t tell us whether the issue landed in the wrong place. Hamel and Shreya’s evals FAQ goes deeper on this.
These examples are made up for the exercises. In your own app, start with the mess you’ve actually got: development failures, user reports, traces you’ve read. If the requirement is already clear, write the check. If two people disagree about what “useful” means, look at examples together first.
Try it: Create .learn-evals/ if needed, then save two observations in .learn-evals/failures.md, including their user impact and proposed checks. Inspect the examples yourself before asking your coding agent to group them. You should be able to point from each proposed check back to the failure that motivated it.
3. Lies, damned lies, and graders
A grader, from lesson 1’s table, checks whether an output meets a requirement. Here it’s a TypeScript function that returns true for pass or false for fail—like a test assertion for the agent’s work.
Run the challenge, then open exercises/grader.ts:
npm run lesson -- 3Expect a table and an intentional nonzero exit. Every challenge in this post prints the same shape: expected is the verdict the example should get, actual is what your function returned, and the third column says whether they agree. So a bad example correctly rejected reads false, false, agreement — we’re grading your grader, not the agent.
Here’s the function you’re repairing:
export function learnerGrader(output: TriageOutput): boolean {
return output.reply.toLowerCase().includes("created");
}It measures whether a word appears. The agent can do fuck-all and still pass. Meanwhile, a valid write followed by “Filed ISS-1” fails.
Find convincing-no-write in the table. The starter accepts that empty tracker because the reply includes “Created.” Your change should reject it using the tracker snapshots.
Start with what changed:
import { newIssues } from "../src/tracker.ts";
const created = newIssues(output.before, output.after);newIssues is a helper in this repo. It finds IDs that weren’t there before. If the tracker started with 20 issues and now has 21, checking for a final total of one would fail a perfectly good write.
Require exactly one new open issue about issue-search, no additional comments, and no changes to existing records. Keep reply wording out of these structural checks. The helpers in src/tracker.ts and reference grader in src/evals/grades.ts are available once you’ve tried an approach.
Don’t just feed it broken outputs. An always-false grader catches every bad result. It also rejects everything that worked. The challenge includes a valid paraphrase so that particular shortcut won’t pass.
Try it: Edit exercises/grader.ts and rerun the same command until every row has graderCorrect: true. Keep the challenge expectations fixed. Explain which false acceptance and false rejection your change fixed. Then inspect npm run examples -- missing-trigger: its issue is structurally valid but loses a reproduction condition. Keep that limitation in mind for lesson 7.
We’re testing the grader here. Later we’ll run the application and use those checks on what it actually does.
4. Build cases that distinguish behavior
Run the lesson instructions:
npm run lesson -- 4It prints a guide rather than scoring anything; your task is to add two contrasting cases to the cases array in src/evals/cases.ts.
A dataset is a collection of cases: inputs, starting state, expected behavior. Open src/fixtures.ts alongside it to see the available starting states. The cases file contains 11 teaching cases and 3 reserved cases. Here are six examples:
| Case | What makes it different | Expected action |
|---|---|---|
new-search |
No existing issue | Create |
true-duplicate |
Matching open issue | Comment on that issue |
similar-title |
Similar wording, different failure | Create |
closed-duplicate |
Matching issue is closed | Create under our policy |
missing-observation |
Actual behavior is absent | Clarify without writing |
search-outage |
Duplicate lookup fails | Explain and don’t write |
Look at true-duplicate next to similar-title. If every case rewards finding a duplicate, an agent that merges vaguely related reports can look great. Give it a case where merging is wrong.
For example, “empty export hangs” and “large export truncates rows” concern the same feature but describe different failures. Conversely, “Esc leaves results narrowed” and “Escape clears the input without resetting the filter” may describe the same failure in different words.
Write the expected behavior before editing the prompt. Keep those expectations outside the input sent to the agent. The model receives the policy and user report, not the answer key.
Try it: Copy an ordinary teaching case as a starting point. Give each new entry a unique id, a behavior slice, a fixture key, the user messages in turns, and an expected action. Leave reserved cases alone. Run npm run check to catch TypeScript mistakes; it won’t tell you whether your expectation makes sense. Finish by explaining why the pair should produce different outcomes—or why different wording should produce the same outcome. We’ll execute cases against the model in lesson 8.
Use the ordinary teaching cases while tuning. The reserved entries are for a later comparison; reading them to design your fix uses them up as unseen examples. A repository your coding agent can inspect is not a secure holdout.
5. Check the process when the process matters
Run the offline challenge and open exercises/trajectory.ts:
npm run lesson -- 5Same table as lesson 3, with the agreement column named passed. The starter misclassifies several paths, so the initial command fails on purpose.
Find read-after-write. The starter sees a read somewhere in the trace and accepts it, even though the comment was made first. Your task is to fix learnerTrajectory so it checks when the evidence became available and which issue the write targeted.
Search returns summaries. Under our policy, the agent has to read the candidate’s details before deciding it’s a duplicate. Saying “I checked the issue” doesn’t count. Neither does calling get_issue before the comment if its result only arrives afterward. The agent needs a successful result before it writes; starting a read is not the same as knowing what it returned.
That doesn’t mean every good run has to use the same script:
| Path | Verdict |
|---|---|
| Search → read target → comment on target | Accept |
| Different search → read another candidate → read target → comment on target | Accept |
| Search → comment → read target | Reject |
| Search → target read fails → comment anyway | Reject |
| Search → start target read → comment → read result arrives | Reject |
| Search → read target → comment call names another issue | Reject |
The read matters. The exact search query usually doesn’t. Anthropic’s agent-evals article covers why checking the whole path too rigidly can reject valid work.
The starter in exercises/trajectory.ts only checks whether get_issue was called. Too late? Wrong issue? Failed call? It waves them through.
The event fields matter here. A tool_call carries id, name, and arguments. The search tool is search_issues; the target of a get_issue call is arguments.id, while add_comment uses arguments.issueId. Its matching tool_result carries toolCallId. Failed results have an error field; successful results have content and no error. The starter comments include this contract so you don’t have to guess it.
Try it: Edit learnerTrajectory in exercises/trajectory.ts. Walk the events in order. Match each tool_result.toolCallId to its tool_call.id. A search counts only after its successful result arrives; the expected issue’s read must also return successfully after that search and before the add_comment call. At that call, require arguments.issueId === expectedIssueId. Then independently check that exactly one comment was stored on that issue, with no new issues or changes to existing records. Both the requested target and the stored target must match. Rerun npm run lesson -- 5 until every row has passed: true, including the rows whose expected verdict is false. Explain one rejected path and one valid alternative.
The exercise teaches a stronger process check than the live suite’s initial search-presence assertion. Once solved, you can explicitly integrate it into duplicate cases in src/evals/triage.eval.ts. Editing an exercise function alone doesn’t change the live suite.
6. Check the whole conversation
Run the next offline challenge and open exercises/conversation.ts:
npm run lesson -- 6Same table again, same intentional failure. Find premature-create-same-ending: the starter accepts the final issue count even though the agent wrote before it had enough information.
You’re repairing learnerConversation. Its output.turns array contains the reply and tracker snapshot after each turn. You don’t need to type follow-up messages into a running agent; these conversations are already constructed for the exercise.
So far: one report, one response. What happens when the report is just “Search is broken”?
The agent needs to ask a question. Follow up with the complete Escape report from lesson 1, and it can create an issue. Follow up with “The issue list. It should work when I use it,” and we still don’t have enough to file.
A final-state check can miss the difference between these two conversations:
| Conversation | After incomplete report | After complete follow-up |
|---|---|---|
| Correct | No write | One issue |
| Premature | One issue | The same one issue |
Both end with one issue. Only one waited for the facts.
That’s why the harness takes a tracker snapshot after every turn. Keep the conversation and tracker alive within the trial. Compare each earlier snapshot with the original state, then check the ending. In a three-turn run, check the middle one too.
Try it: Require turn evidence, reject writes while facts are missing, and retain the final outcome check. Rerun npm run lesson -- 6 until every row has passed: true. Explain how two conversations can end with the same issue but deserve different grades. The supplied cases include continued clarification, intermediate damage later repaired, and a premature middle-turn write.
Those checks can tell us the agent waited. They can’t tell us whether its question helped. “Please provide more details” is pretty useless when the user already gave you those details. Read the reply. Did it ask for the missing fact?
7. Give a judge one job
A model judge is a grader that uses an LLM to apply a written criterion. Our code graders checked counts, IDs, and state. This judge checks whether the stored issue preserves the report’s meaning. First, you’ll make that judgment yourself so you have something to compare it with.
Start by labeling three examples yourself:
npm run lesson -- 7It prints three pairs, each containing id, report, and issue. Read each report next to its proposed issue and decide pass or fail using this one criterion:
Does the stored issue preserve the report’s affected feature, reproduction trigger, expected behavior, and observed behavior without inventing details?
The report is the source of truth. Write down the evidence for each judgment before you see the model’s verdict or the reference labels.
Save your verdicts in .learn-evals/labels.json, replacing each empty string with "pass" or "fail":
{
"pair-01": "",
"pair-02": "",
"pair-03": ""
}Use the IDs your checkout prints. These labels are your comparison data; they never go into the judge’s prompt.
Now we need a model key
We’re using OpenRouter so one API key lets us try different models without changing the code. You can play with free models, though your mileage may vary. A few dollars in credits goes a long way with DeepSeek V4.1 Flash for these small exercises.
Create an OpenRouter key, then copy the example config:
cp .dev.vars.example .dev.varsAdd your OpenRouter key to the ignored .dev.vars file:
OPENROUTER_API_KEY=your-key-here
TRIAGE_MODEL=nvidia/nemotron-3-super-120b-a12b:free
JUDGE_MODEL=deepseek/deepseek-v4.1-flash
JUDGE_PROVIDER=deepseekTRIAGE_MODEL runs the app; JUDGE_MODEL grades its issue text. This config uses a free model for the app and DeepSeek V4.1 Flash for the judge. JUDGE_PROVIDER prefers DeepSeek’s own endpoint, with fallback to another provider serving the same model. To try another model, change its ID and update or clear the provider preference.
Now compare the judge with your labels:
npm run evals:judge -- --labels=.learn-evals/labels.jsonThis makes three judge calls on the fixed examples. No app server needed.
Here’s the summary from my DeepSeek/v2 run against the supplied reference labels, served by DeepSeek. Yours compares against the labels you saved above. Terminal output, excerpted:
┌──────────────┬────────┐
│ (index) │ Values │
├──────────────┼────────┤
│ planned │ 3 │
│ completed │ 3 │
│ errors │ 0 │
│ agreements │ 3 │
│ falseAccepts │ 0 │
│ falseRejects │ 0 │
└──────────────┴────────┘All three calls completed, and all three grades agreed with the reference labels. The command exited with status 0. That’s agreement on three examples, not proof the judge is reliable.
If you get HTTP 429, a call was rate-limited and produced no grade. Count it as an error and check provider availability before rerunning; keep the failed run too.
Open the artifact at the path printed by the command. Completed entries include human, verdict, reason, and quoted evidence; failed calls have an error. A false accept passes an issue you labeled wrong. A false reject fails one you labeled correct. Read the judge’s reasons and quotes when it disagrees with you. Even exit status 0 only means the command ran without errors.
The rubric in src/evals/judge.ts is already v2. Revise it if a disagreement reveals an unclear instruction. If everything agrees, label both batches before adding a borderline pair of your own. Don’t force a disagreement. Stay out of src/evals/recordings.ts while labeling: it contains the reference answers.
Before reading the historical example or reference answers, label the separate validation batch:
npm run lesson -- 7 --validationSave those verdicts the same way in .learn-evals/validation-labels.json, keyed by the IDs it prints. Then run:
npm run evals:judge -- --validation --labels=.learn-evals/validation-labels.jsonKeep that batch out of your rubric edits until you’ve compared the results.
After recording both batches’ labels, run npm run examples -- judge-disagreement for a historical failure. This capture used Nemotron 3 Super as the judge with rubric v1, not the DeepSeek V4.1 Flash/v2 configuration you just ran. Nemotron is also our application’s default model; here it was doing a different job. It accepted an issue with a made-up Redux cause and fix.
DeepSeek agreed with the reference labels on all six pairs across both batches. Those six were inspected while developing the rubric, so they’re practice data — enough to teach calibration, nowhere near enough to certify a judge. Use fresh examples for a real check after tuning.
Also keep three different outcomes distinct:
| Situation | How to treat it |
|---|---|
| Clarification was required, so no issue should exist | Issue-quality criterion is not applicable |
| An issue was required but is missing | Required outcome failed |
| Judge API or response parsing failed | No semantic grade was produced; record an error |
Try it: Explain a disagreement using source evidence, or challenge the judge with a new borderline pair if it agrees on everything. Repeat grading on the same outputs with --repeat=3 to inspect consistency.
8. Run the application repeatedly
So far we’ve fed fixed outputs to the graders. Now we’ll run the issue-triage agent and grade its output.
With .dev.vars configured in lesson 7, start the local Flue application:
npm run devLeave that server running. In another terminal, run one case and inspect its results:
npm run evals -- --cases=new-search
npm run report
npm run evals:reportThe first command runs one live trial, which can pass or fail. A connection error means fix startup; a failed assertion is evidence to investigate, not a result to erase.
npm run report prints counts for the latest run. Compare planned with completed, then check passed, executionErrors, and missingArtifacts. failed covers both assertion failures and execution errors, so open the trial artifact to tell them apart. npm run evals:report prints a URL for a local viewer showing assertions and transcripts; stop it with Ctrl-C before reusing that terminal, and leave the app server running.
The harness creates fresh tracker state, sends the report through Flue’s HTTP interface, waits for completion, and captures the transcript and independent snapshots. Vitest-evals passes that evidence to assertions. Here is a minimal example, also available in examples/article-example.ts:
import { expect } from "vitest";
import { describeEval } from "vitest-evals";
import { createTriageHarness } from "../src/evals/harness.ts";
import { newIssues } from "../src/tracker.ts";
const harness = createTriageHarness({ fixture: "clear-new-report" });
describeEval("filing a new bug", { harness }, (it) => {
it("creates one issue for a complete report", async ({ run }) => {
const result = await run(
"In the issue list, search for a word, then press Escape. " +
"Expected: all issues return. Actual: the input clears " +
"but the list stays filtered until refresh.",
);
const created = newIssues(result.output.before, result.output.after);
expect(created).toHaveLength(1);
expect(created[0]).toMatchObject({
feature: "issue-search",
status: "open",
});
});
});These assertions cover count, feature, and status. The repo’s live suite adds more structural checks. None of them establish that the issue preserved the report’s meaning.
To run this exact example against the app you started, use npm run evals:article. It runs one live trial, which can make several model calls, and writes artifacts/article-results.json. This separate report doesn’t replace the main suite’s results or enter its revision comparisons. Ordinary npm test also executes this same file with a scripted provider and no model API calls.
One result tells you little about consistency. Back to the main suite: run new-search five times.
npm run evals:repeat -- --cases=new-searchRerun npm run report and reopen the viewer. You should have five planned trials to account for, including any that failed to finish. The command prints its workload before starting; five trials keep this small and aren’t a magic sample size. If you want to stay offline, npm run lesson -- 8 shows a synthetic example of mixed results.
Preserve every attempt. If three of five passed, report three of five. Retry-until-green hides failures rather than measuring dependable behavior. You don’t get to edit the fuckups out of the experiment.
pass@k asks whether you get at least one success in k attempts. pass^k asks whether all k succeed. Suppose a case has a constant 80% chance of success and attempts are independent. Three attempts give:
- At least one success:
1 - (1 - 0.8)^3 = 99.2%. - All three succeed:
0.8^3 = 51.2%.
Those are illustrative calculations with explicit assumptions, not estimates from our five runs. Don’t substitute an average across unrelated cases and present it as a measured benchmark. Anthropic explains the reliability distinction.
Try it: Inspect every trial for one case and report how many passed, failed a criterion, or couldn’t complete. Explain why “eventually got one success” would answer a different product question.
9. Make sure trials aren’t contaminating each other
Run the isolation challenge and open exercises/isolation.ts:
npm run lesson -- 9Another offline repair exercise, same table — except that a factory which throws reports error in the actual column instead of a verdict. Expect failed rows and an intentional nonzero exit.
Look at different-trial-starts-fresh. Trial A writes an issue; trial B should start empty, but the starter gives it A’s tracker. The same-trial-retains-writes check can already pass while cross-trial isolation is broken.
In a live eval, that contamination could make trial B discover A’s issue and comment on it instead of creating one. The agent may have followed the policy correctly on the wrong starting state.
Your task is to allocate one tracker per trial ID. Repeated lookups for the same ID must retain its state; different IDs must start from separate copies of the seed. Scope that mapping to the store, so a second store doesn’t inherit the first one’s IDs either.
There’s an easy wrong fix: return a brand-new tracker on every lookup. That stops cross-trial leakage while erasing the conversation history we needed in lesson 6. The checker exercises both requirements, including interleaved writes and closing one trial while another remains active.
Try it: Edit learnerTrialStore in exercises/isolation.ts and rerun npm run lesson -- 9 until every row has passed: true. Explain why both same-trial continuity and cross-trial isolation matter, and why the contaminated comparison needs to be repeated. Compare your solution with the application’s src/trials.ts, which also handles trial expiry and late tool calls.
Shared state isn’t the only way the environment can mess with a score. Timeouts, tool outages, and missing evidence matter too. Anthropic’s infrastructure-noise study shows how execution conditions can move agent eval results.
“Four of four completed trials passed” sounds good. What if ten were planned? Keep both numbers visible. Missing evidence isn’t a pass. Find out why execution failed before changing the prompt.
10. Change one thing and make a decision
Keep the app running from lesson 8, or start it again with npm run dev. This lesson produces a comparison and a written decision about one prompt change. Start with three cases, repeated three times each—nine trials per revision:
npm run evals -- --cases=new-search,true-duplicate,similar-title --repeat=3Inspect the results with npm run report and npm run evals:report. Then make one targeted change in src/policy.ts, restart the application so you’re testing that revision, and run the same command again — eighteen trials across both revisions, several model calls each. Keep models, cases, repetition counts, and graders fixed. Then compare the two archived runs:
npm run compareIt prints the A and B run IDs, then one row per assertion: A and B hold statuses (passed, failed, missing), A_ms and B_ms durations. It picks the latest two readable archives, so check those IDs are the runs you meant. A zero exit means the evidence is all there, not that B won.
Don’t lower the expectations to make the prompt look better. If you discover a genuinely wrong expectation, correct it and rerun both revisions against the corrected criterion.
Here’s an illustrative summary grouping three trials per case; the command prints individual statuses, not fractions:
| Case | Revision A | Revision B |
|---|---|---|
| New report | 3/3 | 3/3 |
| True duplicate | 1/3 | 3/3 |
| Similar title, different bug | 3/3 | 1/3 |
B gets the duplicate right more often. It also merges unrelated bugs. Add up the passes and both revisions score 7/9.
Check correctness before optimizing latency and usage. A judge’s semantic verdict, tool-call count, and response time describe different properties; averaging them together doesn’t make a wrong-target write acceptable. LangChain describes a correctness-first approach to agent comparisons.
Try it: Save a short decision in .learn-evals/decision.md: what improved, what regressed, what remains uncertain, and what the run cost if that information is available. “Unknown” is different from zero, and “inconclusive” is a valid decision. Add a novel case to challenge your conclusion. Keep useful failures as regression cases.
This gives the same checks several possible uses:
| Use | Question |
|---|---|
| Capability eval | Can the application handle a behavior we’re adding? |
| Regression eval | Did a behavior we relied on get worse? |
| Offline comparison | Which revision works better on this controlled set? |
| Production monitoring | Which failures appear in actual use? |
| Shadow evaluation | What would a candidate propose on live inputs with writes disabled? |
| Online A/B test | How do versions affect real outcomes in a randomized experiment? |
These labels tell you why you’re running the eval. They don’t dictate how you grade it. The same state check or calibrated judge can serve several purposes. This repo covers the first three; production monitoring needs real traffic and somewhere to collect it.
What now?
For extraction, check the fields. For retrieval, check whether it found the evidence, then whether the answer used it correctly. With a coding agent, run the code it wrote.
A bigger eval stack might give you dataset storage, annotation tools, experiment tracking, and production integrations. It still needs you to decide what success means and whether the grader can catch a known failure. You don’t have to relearn those decisions because you started with Vitest.
Ask the teaching skill to apply this to my app, or pick one real failure yourself. Save the evidence. Write down what should have happened. Make the check reject that failure and accept something valid. Then change the app and run it again.
For more reading, grab the agent evals reading pack. Start with the top three, or give the pack to your coding agent and ask it to walk you through one of the articles.