Tag: workflows
Posts tagged workflows:
-
Simple vs Plan: When to Use Which (AppWorld smoke cohort) — Simple vs plan on AppWorld skills smoke: plan wins some tasks, ties on others, and always costs more. Choose the mode that fits the job — both belong in the toolkit.
-
Multi-Agent vs Single-Agent: When Planning Beats Reacting (MCP Tool Tax + pass@k) — Single-agent MCP loaded 453 tools (~17k peak). A planner peaked lower, spent ~2× total tokens, and sometimes won quality. Report pass@1 and pass@3 — that is not pass^k.
-
How to Evaluate AI Agents: Ban Clarifying Questions and Zero-Tool “Success” — Unattended AI agent evaluation fails when the model asks the user or reports done with zero MCP calls. Put those rules in the harness, not the framework.
-
Running AppWorld Locally for Aiden Agent Evals: Docker Compose, MCP, and Things We Wish We Knew Upfront — Run AppWorld locally with the Aiden agent runtime: docker-compose, MCP HTTP wiring, health gates, and lessons from replacing custom shims with the official three-server stack.
-
Stop Spawning Duplicate Workers: What a Handoff Gate Changed in Agent Evals — Stop duplicate agent workers: a handoff gate fixed fairness 5/5 and cut planner token tax on AppWorld evals — partial pass% rose while strict TGC remains a separate harness goal.
-
AI Agent Eval Failure Modes: Budget, PII Placeholders, and Self-Reported Passes — AI agent eval failure modes on AppWorld: budget stops before evaluate, PII placeholders poison tool calls, 422 wrong methods, and prose PASS vs judge FAIL.
-
Agent Orchestration Tax: Tokens, Iterations, and Tool Calls After a Fair Eval — Agent orchestration tax after a fair eval: on AppWorld tasks, planner paths cost ~1.6× tokens and ~3× iterations — measure coordination cost separately from benchmark success.
-
Fair Agent Evals: Don’t Compare Planner vs Single-Agent Until Tools Match — Fair agent evals for planner vs single-agent: match domain tool access on workers before you compare tokens, pass rate, or declare a routing winner.
-
Single-Agent vs Multi-Agent Orchestration: How to Choose — Single-agent vs multi-agent for SRE triage: a fair A/B, what each shape wins at, and a decision framework so you stop defaulting to either.
-
Aiden the Easy Way: One Module from Vague Issue to Review PR — Aiden the easy way: one OpenTofu module from a vague GitHub issue to a review PR — Specify, Research, Plan, without wiring every resource by hand.
-
Aiden the Hard Way: Vague GitHub Issues to Review PRs — Aiden the hard way: turn a vague GitHub issue into a review PR — GitHub integration, agent, workflow, webhook, and status poll wired by hand.
-
From Demo to Deploy — Failure Modes with Receipts — From demo to deploy: production-ready AI agents need receipts, not fluent demos — evidence gates, HITL tiers, and eval checklists for enterprise pilots.
-
The Diary Learning Loop — From Daily Agent Digests to Human-Approved Policy — AI agent learning loop: daily digests become human-approved workflow and policy changes — not a bigger vector store.
-
Your RCA Agent Doesn’t Need Another Runbook — It Needs a Map — Your RCA agent doesn’t need another runbook — it needs a map. Topology, gates, and verify-first navigation beat a forty-page notebook.
-
Evidence-Gated RCA — Prove, Then Narrate — Evidence-gated RCA for AI SRE agents: prove with receipts, then narrate. Fixed stages, structural evals, and token-aware tool loops.