What is the best way to debug a multi-step AI agent?

Freeze one golden prompt you can rematch cold, export one execution bundle for that run, grade what the model saw on each step, separate tools that paid rent from waste, then change one host gate or tool contract. Rewrite the system prompt only after the harness is honest.

Key takeaways

  • One prompt + one zip beats prompt thrash.
  • Grade observations and tool outcomes per step.
  • Fix host contracts before prose.

When a multi-step agent gives a plausible but unsupported incident diagnosis, start from the execution record rather than rewriting its prompt. This procedure uses Aiden traces as an example, but applies wherever model calls and tool observations can be exported. For multi-stage bring-up, see debug agent workflows like hardware.

What the evidence supports

  1. One golden prompt you can re-run cold.
  2. One debug zip / session export (one zip, one conversation).
  3. Grade per step: what the model saw, which tools paid rent, which were waste.
  4. Fix host contracts before prose.
  5. Rematch the golden prompt and publish absolute wall/tokens/correctness.

Step 1 — Freeze the errand

A two-part test—alert triage plus a log comparison, as in our bench—makes omissions visible. Freeze the request, expected evidence, time windows, and tool access; otherwise a rematch may test a different task. A single-part job can still work if its expected observations are explicit.

Record: model, orchestration shape (single-agent ReAct vs hierarchical ReAcTree — primer · PDF), host flags, tool endpoint, session id.

Step 2 — Export once

Prefer a single artifact: transcript + tool results + stage timeline. If you only have logs, at least pull:

  • elapsed seconds
  • token in/out
  • tool names + error strings
  • final Theory / Unknowns / Do-this-now
  • per model call: model name, whether the response id starts with resp_ (Responses API) or chatcmpl_ (Chat Completions), and whether usage lists reasoning tokens (Completions vs Responses)

For hierarchical runs, merge parent and child spans (timed records of model and tool calls) using the session id. A parent-only view can miss a child’s measurement or hide a child’s repeated failed query.

Step 3 — Inspect observations and claims

Freeze a task, export the trace, find the faulty step and replay it

Change one host or tool behavior at a time so the next run is a useful comparison.

Read the sequence from tool request to observation to final claim. For example, an HTTP 200 may carry a query envelope whose outcome is failed; transport success is not measurement success. Ask four questions:

Question Fail looks like
Did measurement tools run? Only search_tools / skills
Did failures look like failures? HTTP 200 empty treated as success
Did observations change? Same failed LogQL five times
Did the close name receipts? Undetermined with no blocked query

Bring-up multi-stage workflows like hardware: green one stage at a time.

Step 4 — Change the host

Change one variable at a time. For a repeated failed query, this order helps distinguish a broken tool contract from a model instruction problem:

  1. Typed failure / no-progress / fan-out (stop retrying)
  2. Tool path contracts (cursor paging for truncated tool output)
  3. Prefer parent measurement before children (single vs multi)
  4. Prompt / skill text last

Mid-run steer belongs here too (steer agents mid-run) — operators need a cancel path when the loop is clearly stuck.

Step 5 — Rematch with absolute measures

If wall doubled, say wall doubled (relative efficiency lies). If correctness jumped on one seat, show the Theory text.

Keep both before-and-after traces. If correctness improved but the tool server was less contended, do not attribute the wall-time change to your code. If the new gate halts an agent early, require it to carry the partial evidence and blocked query into the final answer.

FAQ

What is the best way to debug a multi-step AI agent?

Freeze one golden prompt, export one execution debug bundle, grade what the model saw per step, separate tool good-vs-waste, then change one host gate or tool contract — not the system prompt first.

Should I start by rewriting the system prompt?

No. Prompt changes hide harness bugs. Fix typed failures, fan-out, and completion gates first; use prompts for role and output shape only.

What artifacts do I need?

Session id, elapsed seconds, token totals, tool call list with outcomes, final Theory text, and ideally a single zip that holds the conversation plus tool payloads. To see which OpenAI API and how much thinking ran, check whether response ids start with resp_ or chatcmpl_, and whether usage lists reasoning tokens.

What are the best practices for debugging complex agentic workflows?

Bring up one stage at a time against a golden gate, score committed tool effects rather than raw transcripts, and keep a single export per conversation so nested chains stay inspectable.