A model-card score does not tell an operator how long an alert-and-logs investigation will take with a particular tool server. We ran the same task across six model-and-orchestration combinations and recorded the answer quality, wall time, tokens, and estimated or traced cost.

We ran one dual-part observability job six ways:

  1. xAI-class reasoning × single-agent / hierarchical
  2. OpenAI gpt-5.4 (generate) × single-agent / hierarchical
  3. OpenAI Responses reasoning preview × single-agent / hierarchical

Same live Grafana/Loki plane. Same prompt. Same host settings. We ran this on our production agent runtime (Aiden). This is the scorecard. Qualitative close styles live in how the models think. Tool inventory lives in the tool menu.

There was one run per cell in each wave. The rankings below describe those runs only; repeat trials would be needed to estimate variability or select a production default.

Shapes we compared

Fair comparison holds task and tool conditions steady while comparing outcomes

The benchmark compares models and execution shapes under the same task and tool access.

Two orchestration shapes. First mention definitions (used everywhere below):

Shape Open name Meaning
single-agent Single-agent ReAct loop One agent, reason→act→observe in one trajectory
hierarchical Hierarchical ReAcTree planner Parent decomposes subgoals, may spawn child agent nodes, control-flow coordination

Background: What is ReAcTree? · paper PDF / abs · production notes ReAcTree bugs.

What the evidence supports

  • Job: triage a sustained goroutine-growth alert, then compare 24h vs 7d logs in one Kubernetes namespace for anomalies and cross-links.
  • Highest cohort-relative score (wave 1): xAI hierarchical — ~141s, ~191k tokens, corr 1.0, efficiency 1.375.
  • gpt-5.4 hierarchical matched quality with ~255k tokens vs ~373k for single-agent (~0.68× tokens — smaller total context bill across the whole tree).
  • Responses reasoning preview burned ~475–516k tokens; hierarchical once scored 0.69 correctness (thin Undetermined close).
  • Efficiency without absolute wall is a trap — see relative scores.

The job (obfuscated)

Part A. A firing alert reported sustained growth in go_goroutines, a Go runtime gauge of active goroutines. Its 24h derivative estimates the rate of change; the alert rule’s for duration requires the condition to persist for hours. The agent had to inspect the rule and affected service instances, then write a hypothesis (Theory), unresolved questions (Unknowns), and an immediate next action.

Part B. In the same session, query Loki (the log backend) for Kubernetes namespace ops-platform. Compare the last 24h with a 7d baseline for message volume, errors, and new signatures. The agent should say whether the logs support Part A; a query failure must be named as an Unknown, not converted into evidence of absence.

Correctness = 13-point checklist (Theory, Unknowns, next action, measured alert, numbers, namespace, both windows, compare language, logs/Loki, honesty, Part A/B labels).

Efficiency =

correctness / (0.45·wall/median + 0.45·tokens/median + 0.10·cost/median)

Cost: from session traces where present; otherwise estimated using the gpt-5.4 $/token blend for that wave. The estimates are not provider invoices, and their cross-model rank is less secure than the wall and token columns. The formula also divides by medians from the current cohort, so its score cannot be compared directly across waves.

Wave 1 leaderboard

Rank Combo Shape Wall Tokens Cost Corr Eff
1 xAI · hierarchical ReAcTree 141s 191k ~$0.16 est 1.00 1.375
2 xAI · single-agent ReAct 157s 187k ~$0.15 est 1.00 1.301
3 gpt-5.4 · hierarchical ReAcTree 148s 255k $0.37 1.00 1.107
4 gpt-5.4 · single-agent ReAct 154s 373k $0.30 1.00 0.935
5 Responses · single-agent ReAct 285s 475k ~$0.39 est 1.00 0.617
6 Responses · hierarchical ReAcTree 200s 516k ~$0.42 est 0.69 0.480

Hierarchical vs single-agent ratios (hierarchical / single-agent):

Model class Wall Tokens Quality
xAI 0.90× ~1.02× both full
gpt-5.4 ~0.96× 0.68× both full
Responses preview 0.70× 1.09× hierarchical collapsed

What moved when we added no-progress gates

Second wave: same six cells on a harness that already preferred parent measurement before spawning children, plus typed query failure handling, same-observation halt, and same-tool fan-out caps (stop retrying dead probes).

Absolute changes matter more than efficiency deltas:

Combo Wall × Tok × Corr
xAI hierarchical 1.86× ~1.01× 1→1
xAI single-agent 1.74× 1.27× 1→1
gpt-5.4 single-agent 1.72× 1.49× 1→1
gpt-5.4 hierarchical 1.91× 1.88× 1→1
Responses hierarchical 2.80× 1.26× 0.69→1.00
Responses single-agent 2.22× 1.23× 1→1

Wave 2 also kept six Grafana clients active in parallel for the full run. That makes the wall-time ratios unsuitable as isolated estimates of the gates’ effect. The Responses hierarchical run changed from a 0.69 stub to a full Part A/B close, but one rematch cannot attribute all of that improvement to the gates alone.

Monday rules

  1. Publish wall, tokens, cost, correctness as four columns. Label estimated cost and cohort-relative efficiency separately.
  2. Never crown a model on one shape. Single-agent ReAct and hierarchical ReAcTree are different products.
  3. Test reasoning-preview seats behind the same completion and budget gates; in these runs they consumed ~475–516k tokens in wave 1, but a different incident or effort setting may behave differently.
  4. When a harness change lands, re-run the same prompt. Without a rematch, a before/after story cannot separate the change from the task and tool conditions. Random one-incident eval reruns are the same trap with extra models (same incident, five models).

Related: What is ReAcTree?, hierarchical vs single-agent on this job, hybrid plan roster, AI SRE benchmarks, reasoning effort is not free.

FAQ

What did you hold constant across the six combos?

Same dual-part prompt (firing alert triage + namespace log anomaly 24h vs 7d), same live Grafana/Loki tools, same host settings, same wall budget. Only model class and orchestration shape (single-agent ReAct vs hierarchical ReAcTree) changed.

Which combo won on efficiency?

xAI-class reasoning under hierarchical ReAcTree ranked first on both harness waves. OpenAI gpt-5.4 followed. OpenAI Responses reasoning preview was slowest and most expensive per useful close.

How do you define efficiency?

correctness / (0.45·wall/median + 0.45·tokens/median + 0.10·cost/median). Relative within a cohort. Always publish absolute wall and tokens next to it.

Did hierarchical ReAcTree beat single-agent ReAct?

On this job, hierarchical was a slight win for xAI (faster wall, similar tokens). For gpt-5.4, the hierarchical run used fewer total prompt tokens in the first wave. For the Responses reasoning preview, hierarchical once collapsed quality and later recovered only after host gates — never the cheap win.