AI Agent Runtime: What to Measure Before You Buy

You already know the definition of an AI agent runtime. This page is the buyer’s receipt: what to demand on a golden prompt before you trust a demo.

TL;DR

A runtime is not “LLM + tools.” It is the loop with teeth: budgets, typed tool outcomes, completion gates, and an eval you can re-run.

Before you buy (or build), demand numbers on your golden prompt. Ours: six seats, one alert+logs job (scorecard). Wall ranged ~141–285s in wave 1; one hierarchical ReAcTree seat scored 0.69 correctness until host gates improved the close.

Shapes: single-agent ReAct loop vs hierarchical ReAcTree planner (what is ReAcTree? · PDF).

Explain like I’m five

Buying a race car from a brochure is silly. Ask them to drive your driveway, time the lap, and show what happens when a tire is flat.

Five questions for any runtime vendor

Show the loop. Where do tool results re-enter context? What gets truncated and how do you page it back?
Show failure classes. Does empty PromQL look like success?
Show the done button. Completion gate, not “model said finished.”
Show single-agent and hierarchical on the same prompt (merits).
Show absolute wall/tokens/cost/correctness — not a relative badge (efficiency trap).

What good looks like on a triage job

Theory / Unknowns / Do-this-now with receipts (RCA eval)
Measurement tools dominate catalog tools (tool menu)
Dead queries halt (no-progress)
Operator can steer mid-run

What bad looks like

Demo on a mocked tool that never returns failed
Only one model × one orchestration shape published
Prompt-only “don’t retry” instructions
Multi-agent diagrams with no orchestration tax numbers

If you are building in Go, start from why Go and the runtime definition. If you are buying, bring this page to the sales call and ask them to drive your driveway.

FAQ

What is an AI agent runtime?

The production loop that turns goals into tool calls and durable state: model routing, tool contracts, budgets, completion gates, and audit — not a notebook demo or a single chat completion.

What should I measure before buying an agent runtime?

On your golden prompt: wall time, tokens, cost, checklist correctness, tool good-vs-waste, and behavior when queries fail. Ask for a rematch after they change models — and across single-agent ReAct vs hierarchical ReAcTree if both are offered.

Why do model cards fail as a buying guide?

The same prompt across six model×orchestration seats produced a 2× wall spread and a hierarchical correctness collapse on one reasoning preview. The runtime’s gates mattered as much as the model name.