Agent evals starter pack
A short path for engineers putting agent evals in CI and growing coverage without treating draft PRs as closure.
The five posts
- The AI Agent Testing Pyramid. Fast bounds, contract checks, a few live trials.
- AI Agent Evaluation: Why the Grader Must Be Separate. Agent model is not the judge.
- Supervised E2E Learning Loop. Automation drafts; humans merge, deploy, re-assert.
- When Is an E2E Finding Actually Closed?. Scope, disposition, runnable asserts, real execution.
- E2E Learning Loop Blind Spots. Skip-as-success and other bookkeeping failures.
Next
- Hub: AI agent workflows
- Related: Two E2E Feedback Loops · Deterministic Checks vs LLM-as-a-Judge · AI Agent Evals in CI/CD
- Full series: Building an Enterprise AI Agent Platform in Go
- Service path: SRE as service starter pack
Stay in the loop — production notes on AI agents, workflows, and SRE.
Low volume — new posts and curated reading lists. Unsubscribe anytime.
FAQ
Where should I start on AI agent evals?
Read the testing pyramid first (what belongs in CI), then why the grader must be separate from the agent. After that, the supervised E2E loop posts cover how suites get stronger without auto-merge theater.
How long is this pack?
Five posts. TL;DRs fit in one sitting; the E2E closure and blind-spot posts repay a careful read if you run live trials.