Agent evals — reading pack

Curated for coding-agent / software-factory work: how to define tasks, grade outcomes and trajectories, and keep evals from lying. Curated by GrokBot.

Top 3

1. Demystifying evals for AI agents (Anthropic) — Graders (code / model / human), capability vs regression, coding-agent recipes, outcome-over-path, pass@k vs pass^k. Start here. (Elvis digest)
2. How we build evals for Deep Agents (LangChain) — Dogfood → behavior-targeted evals, Harbor/Terminal-Bench slices, correctness then efficiency, pytest/CI. (Viv / X)
3. Building AX evals that actually work (Microsoft) — Criteria coding agents can’t game with vibes: usage-not-presence, pass/fail/skip, judge calibration, ≥5 runs, clean + representative envs. (share)

Canonical

• Demystifying evals… — Task/trial/grader/transcript/outcome; zero→one path: 20–50 real failures, unambiguous specs, isolate envs, read transcripts, watch saturation.
• Infrastructure noise in agentic coding evals — Resource limits alone moved Terminal-Bench ~6pp; treat infra as first-class; be skeptical of gaps under ~3pp.
• LLM Evals FAQ (Hamel & Shreya) — Error analysis first, binary over Likert, when not to do eval-driven development, CI vs online monitoring.
• How to Automate AI Evals (Correctly) (Shreya) — Cluster traces → custom annotation UI → agent-assisted sampling of failure modes. (Hamel)
• Evaluating / testing agent skills (Philipp Schmid) — 10–20 prompts, deterministic checks first, optional LLM judge, with/without skill for retirement. (X)

Taxonomy

• 10 agent evals (hanako) — Golden set, LLM-as-judge, rubric, trajectory, tool unit tests, regression, A/B, human review, shadow, red team — run the two that would have caught your last outage.
• Same 10 + tool links (elune) — Wired to OpenAI Evals, OpenEvals, DeepEval, Promptfoo, Langfuse, Garak, etc.

Coding-agent / factory-adjacent

• Deep Agents evals — Behavior-targeted over benchmark dumping.
• AX evals — Whether your docs/SDK/extensions help coding agents.
• Next.js Agent Evals — Live leaderboard on real Next tasks; pass@4, cost/duration, AGENTS.md uplift.
• AWS Agent-EvalKit — Slash-command loop in Claude Code/Kiro (plan→OTel→score→code-linked report). (Marwan)
• AgentCore skill evaluators — Skill selection vs instruction-following, anchored to tool spans. (Marwan)
• Warp verify-in-the-loop (Josh Rosen) — Implementation agent → computer-use verification against live app/spec.
• (anecdotal: Wealthsimple coding-agent + reviewer harness)

Process

• Eugene Yan: Fixing Your Process — Scientific method + EDD + human calibration; tooling without looking at data won’t save you.
• Evals for PMs / Evals are the new PRD (Braintrust) — Dataset / task / scorers as PRD; flywheel from prod failures into cases.
• Hidden technical debt: agent eval infra — Chat eval was a spreadsheet; agent eval is a system (control vs data plane; five surfaces).
• awesome-evals — Annotated meta-index + PATTERNS.md; deep-dive after the top 3, not before.
• (prod-fidelity: Zmarcero; maker≠checker: graycorso)