If you are trying to evaluate AI agents on a tool-using benchmark, the first question is not “which mode wins?” It is whether both modes could use the same tools on the worker that does the work.

We ran paired evals on AppWorld — a controllable multi-app benchmark (paper) — through our agent runtime in Aiden, with domain APIs exposed via Model Context Protocol (MCP). The early numbers lied. The planner path looked cheaper until we realized workers often had zero domain API calls while the single-agent loop was doing the job.

This post is part one of a three-part series on fair agent evals: fix tool parity, then measure orchestration tax, then classify failure modes. Part two: orchestration tax. Part three: failure modes. Sequel: handoff gate.

AppWorld agent eval series: fair evals, orchestration tax, failure modes, handoff gate

Fair agent evals: single-agent with tools vs planner worker without domain MCP access

What this series measures (and what it does not)

This is a methodology series about how to run fair planner-vs-single-agent evals — not a claim that AppWorld scores predict production outcomes.

Lens What we measured What Aiden ships in production
Benchmark AppWorld — multi-app API mutation tasks (phone, payments, notes, …) SRE triage, evidence-gated RCA, operator workflows
Goal of the run Calibrate fairness gates, orchestration tax, failure classes Incident response, diagnostics, audit trails
Sample size 5–10 tasks per cohort (directional) Customer environments with different SLOs

On these small AppWorld slices, strict TGC/SGC success (judge.success) did not clear — both modes often landed at 30–50% partial pass_percentage with harness blockers (budget, API discovery, PII redaction in eval mode). That tells us where the eval harness still needs work. It does not mean the runtime fails at jobs it was built for. When we say “judge did not pass,” read it as “this benchmark bar on this slice” — the same way a unit test red does not mean the whole product is broken.

TL;DR

  • Unfair eval: planner workers never received the domain MCP pack → “cheaper” planner, 0 domain calls, hallucinated success in transcripts.
  • Fairness gate: after handoff fix, 10/10 pairs passed tool-access checks on a mixed ten-task cohort; workers averaged ~20 domain calls vs ~14 on single-agent — both sides actually ran the benchmark.
  • Benchmark bar: strict AppWorld TGC did not clear on this slice (both modes); partial pass_percentage often 30–50% — agents did real API work, harness still blocked full success.
  • Monday-morning rule: do not publish planner vs single-agent numbers until the worker registry shows the same domain tools you expect on the root in simple mode.

Explain like I’m five

Comparing two cooks is unfair if one gets ingredients and the other only gets a phone to call a helper who was never given a kitchen key. The helper might write a beautiful note saying dinner is ready. You still have no food.

What we were actually comparing

Two execution shapes on the same AppWorld tasks:

Shape Who calls domain APIs Typical orchestration
Single-agent loop Root agent holds the full MCP tool pack One plan → tool → context cycle
Planner + worker Meta root delegates; worker should hold the MCP pack Spawn subcontractor with explicit tool_names

The benchmark judge is AppWorld’s own evaluate step (their TGC/SGC — task goal completion / scenario goal completion). We treat that as ground truth, not assistant prose. Telemetry came from Langfuse generation counts (aggregates only in this write-up).

This is the eval sequel to single-agent vs multi-agent orchestration — same “match the axes” discipline, applied to a public tool benchmark instead of a single-plane triage dig.

The unfair run (why the planner looked cheap)

On an early three-task smoke, averages looked like this:

Mode Avg domain API calls Avg worker spawns What actually happened
Single-agent ~14 0 Linear load → login → call → evaluate
Planner 0 ~1.3 Root spawned workers; workers often had no domain tools

The planner path burned tokens on create_agent / search_tools while never calling the apps under test. In at least one run the transcript claimed export success and evaluation pass with no tool receipts — classic self-report without a judge call.

Interpretation: that is not “orchestration is cheaper.” That is an invalid A/B.

Eval pipeline: fairness gate before comparing tokens or modes

Bar chart: domain API calls per run before handoff fix (single-agent ~14, planner ~0) vs after fix on n=10 cohort (13.8 vs 20.5)

Caption: After the handoff fix, both paths call domain APIs. Strict TGC on this slice: see benchmark context above.

The fairness gate (what “fair” means here)

We defined tool-access fairness separately from task success:

  1. Single-agent: domain MCP tools present; no worker spawn tool on the root.
  2. Planner: at least one worker spawn; worker registry includes the full domain MCP pack; no hard-fail because optional infra names were missing from the registry.
  3. Unattended: no clarify / permission prompts (eval mode must not wait for a human).
  4. Judge: evaluate result parsed; fallback DB scoring labeled when the agent stopped on budget.

On a two-task fair canary, 2/2 pairs passed fairness. Domain calls averaged 12 (single-agent) vs 31.5 (planner workers) — both sides touched the APIs.

On a ten-task mixed cohort (simple-band and advanced-band tasks), 10/10 pairs passed fairness. Averages: 13.8 vs 20.5 domain calls, 14.6 vs 43.8 model iterations, ~1.6× tokens. Neither mode cleared strict AppWorld TGC on this harness slice — expected while budgets, discovery, and eval redaction policy are still being tuned.

Fairness does not mean matched compute budget (planner runs allowed slightly higher caps and worker node limits in this harness). It means both sides could do the work.

Copy-paste fairness scorecard

Before you compare planner vs single-agent on any tool benchmark, record:

# Check Pass?
1 Worker (or root) registry lists every domain tool the task needs  
2 Planner root is not required to call domain APIs if design is meta-only  
3 Spawn payload does not hard-fail on benign extra tool names  
4 Parent telemetry does not count subagent tools as root tools (or you split roles in analysis)  
5 Unattended mode — no HITL / clarify stalls  
6 External judge invoked and parsed; prose PASS ≠ judge PASS  

If row 1 fails, stop. Fix wiring, then rerun. Tokens and “wins” before that are noise.

Lessons learned

  1. Cheaper can mean “did not run the benchmark.” Zero domain calls is an abort signal, not a routing win.
  2. Fairness is a gate, not a score. Passing tool-access checks is prerequisite to comparing modes — it does not by itself clear a hard multi-app benchmark bar.
  3. Handoff bugs look like mode failures. “Planner has no APIs” was registry wiring, not a law of trees.
  4. Self-report is not evaluate. Fluent PASS lines without judge receipts are a failure class, not a tie-breaker.
  5. The pattern generalizes. Any planner-worker runtime needs the same checklist before you trust mode comparisons.

On this site

Elsewhere

Acknowledgments. Built with the StackGen Aiden team — the engineers behind the agent runtime and platform this series describes.

🚀 We’re building AI-powered SRE at StackGen. If you’re tired of 3 AM pages and want AI agents that triage incidents, run diagnostics, and draft RCA reports — check out ai.stackgen.com and try our new SRE offering.

FAQ

What makes an AI agent evaluation fair when comparing planner vs single-agent?

Both paths must reach the same domain tools on the worker that actually mutates state. The planner root may stay meta-only, but the subcontractor must get the full MCP pack. Also match unattended mode, judge parsing, budgets, and record whether tools were hard-denied vs soft-dropped.

Why did our planner look cheaper before we fixed the handoff?

The planner path spent budget on spawn and repair while workers called zero domain APIs. That is a wiring bug, not evidence that orchestration is efficient. Cheap runs that never touch the benchmark APIs are invalid comparisons.

How do you evaluate AI agents that use MCP tools?

Log tool-access parity first: which names were on the root vs worker registry, whether create-worker calls hard-failed on missing names, and whether parent telemetry double-counts subagent calls. Only then compare tokens, iterations, and external judge pass rate.

Does AppWorld judge pass mean the agent succeeded?

AppWorld publishes TGC/SGC through its own evaluate harness — not your runtime's self-report. Strict success is a high bar on multi-app mutation tasks. This series uses AppWorld to calibrate eval fairness and failure taxonomy; it is not a scorecard for Aiden's production SRE workflows.

Is this only about one agent runtime?

No. Any planner-worker shape — LangGraph, CrewAI, AutoGen, OpenAI Agents, or an in-house tree — needs the same fairness gate before you compare modes.