Tag: ai-agents

Posts tagged ai-agents:

• Simple vs Plan: When to Use Which (AppWorld smoke cohort) — Simple vs plan on AppWorld skills smoke: plan wins some tasks, ties on others, and always costs more. Choose the mode that fits the job — both belong in the toolkit.

• Multi-Agent vs Single-Agent: When Planning Beats Reacting (MCP Tool Tax + pass@k) — Single-agent MCP loaded 453 tools (~17k peak). A planner peaked lower, spent ~2× total tokens, and sometimes won quality. Report pass@1 and pass@3 — that is not pass^k.

• How to Evaluate AI Agents: Ban Clarifying Questions and Zero-Tool “Success” — Unattended AI agent evaluation fails when the model asks the user or reports done with zero MCP calls. Put those rules in the harness, not the framework.

• Running AppWorld Locally for Aiden Agent Evals: Docker Compose, MCP, and Things We Wish We Knew Upfront — Run AppWorld locally with the Aiden agent runtime: docker-compose, MCP HTTP wiring, health gates, and lessons from replacing custom shims with the official three-server stack.

• Stop Spawning Duplicate Workers: What a Handoff Gate Changed in Agent Evals — Stop duplicate agent workers: a handoff gate fixed fairness 5/5 and cut planner token tax on AppWorld evals — partial pass% rose while strict TGC remains a separate harness goal.

• AI Agent Eval Failure Modes: Budget, PII Placeholders, and Self-Reported Passes — AI agent eval failure modes on AppWorld: budget stops before evaluate, PII placeholders poison tool calls, 422 wrong methods, and prose PASS vs judge FAIL.

• Agent Orchestration Tax: Tokens, Iterations, and Tool Calls After a Fair Eval — Agent orchestration tax after a fair eval: on AppWorld tasks, planner paths cost ~1.6× tokens and ~3× iterations — measure coordination cost separately from benchmark success.

• Fair Agent Evals: Don’t Compare Planner vs Single-Agent Until Tools Match — Fair agent evals for planner vs single-agent: match domain tool access on workers before you compare tokens, pass rate, or declare a routing winner.

• From Vibes to Contracts: How We Rebuilt Agent Evals Around an Industry Standard — From vibes to contracts: how we rebuilt agent evals around eval sets, rubrics vs criteria, a grader stack, and pass^k reliability.

• Canary First: Lessons from Black-Box Consistency Evals for Live SRE Investigate — Canary-first evals for live SRE investigate: check the canary before burning judge tokens, and never treat a draft RCA as done.

• AI SRE Agent Benchmarks: Wall Time, Tool Calls, Tokens, and ReAcTree Tax — AI SRE agent benchmarks: wall time, tool calls, tokens, and ReAcTree tax — fair A/B numbers so you know when orchestration is worth the cost.

• Reasoning Effort Is Not a Free Upgrade for Tool-Heavy Agents — Reasoning effort is not a free upgrade for tool-heavy AI agents. Live SRE A/Bs: blanket high timed out, adaptive low→high dug deeper.

• Claim-Aware Evidence Packing — Don’t Accuse Agents of Inventing What You Truncated — Claim-aware evidence packing: don’t accuse AI agents of inventing what you truncated. Why hallucination guards fail when they drop the receipts.

• Single-Agent vs Multi-Agent Orchestration: How to Choose — Single-agent vs multi-agent for SRE triage: a fair A/B, what each shape wins at, and a decision framework so you stop defaulting to either.

• Aiden the Easy Way: One Module from Vague Issue to Review PR — Aiden the easy way: one OpenTofu module from a vague GitHub issue to a review PR — Specify, Research, Plan, without wiring every resource by hand.

• Aiden the Hard Way: Vague GitHub Issues to Review PRs — Aiden the hard way: turn a vague GitHub issue into a review PR — GitHub integration, agent, workflow, webhook, and status poll wired by hand.

• What Are SRE AI Agents? — What are SRE AI agents? AI for incident triage, diagnostics, and RCA with bounded autonomy — not a chatbot and not open-ended remediation demos.

• What Is an AI Agent Runtime? — What is an AI agent runtime? The production loop that plans, calls tools, and manages context — not a GenAI platform, chatbot, or notebook demo.

• SRE for Agentic Systems: Why Uptime Isn’t Enough Anymore — SRE for agentic systems: uptime isn’t enough. Judgment SLOs and how to measure agentic drift when the agent can be ‘up’ and still wrong.

• PII Redaction for AI Agents — Two Views, One Trace — PII redaction for AI agents needs two views: a protected model history and authorized operator visibility when debugging tool calls.

• How to Steer an AI Agent Mid-Run Without Starting Over — Can you interrupt or redirect an AI agent mid-response? Yes. How to send steer or cancel signals mid-stream and change the active task without restarting.

• Prompt Caching for AI Agents Is an Architecture Problem — Prompt caching for AI agents is an architecture problem: stable prefixes, early compaction, and references beat copying a turbulent payload.

• AI Agent Loop Detection — Don’t Throw Away the Answer — AI agent loop detection can erase a good answer. Preserve the best evidence-backed result when a stalled run ends — don’t throw the work away.

• AI Agent Root Cause Analysis — Evidence Discarded After the Lead — AI agent root cause analysis fails when a dig finds a lead and discards it — peer noise, wrong fire-time windows, and transcript gates for AI SRE.

• Be Creative. Don’t Invent. — When an AI SRE agent hits a dead end, be creative — don’t invent. Search harder instead of hallucinating rule IDs, metrics, or a tidy RCA.

• AI Agent Root Cause Analysis — Curiosity Before Confidence — AI agent root cause analysis for SRE: curiosity before confidence. Soft prompts don’t stop bad RCAs — checklists, hard gates, and batched validation do.

• Is the Task Actually Done? — Completion Loops for Production Agents — Is the AI agent task actually done? Why production agents need an independent completion check — not a self-graded ‘I’m finished.’

• When Your AI Agent Scorecard Lies — When your AI agent scorecard lies: measure telemetry quality before you trust reliability, correctness, cost, or latency scores.

• The Hypothesis Ladder — Ruling Things Out Before You Narrate — Hypothesis-driven AI SRE root cause analysis: climb identity and onset before deploy theories, keep parallel branches, prove first and narrate last.

• From Demo to Deploy — Failure Modes with Receipts — From demo to deploy: production-ready AI agents need receipts, not fluent demos — evidence gates, HITL tiers, and eval checklists for enterprise pilots.

• The Diary Learning Loop — From Daily Agent Digests to Human-Approved Policy — AI agent learning loop: daily digests become human-approved workflow and policy changes — not a bigger vector store.

• Beyond Confluence Runbooks: Why GitOps Triage Steps Matter in the AI Era — Beyond Confluence runbooks: why GitOps triage steps matter for AI agents — version-controlled procedures that change with your stack.

• Your RCA Agent Doesn’t Need Another Runbook — It Needs a Map — Your RCA agent doesn’t need another runbook — it needs a map. Topology, gates, and verify-first navigation beat a forty-page notebook.

• LLM Tokenomics for Production Agents — Context Budgets as an Operating Model — LLM tokenomics for production agents: token budget strategies, context compression, and FinOps loops that keep sessions finishing.

• Evidence-Based Verification — Don’t Trust Self-Report, Check the System — Evidence-based verification for AI agents: don’t trust self-report — pull proof from ArgoCD, Datadog, and systems of record, then let Go own pass/fail.

• Evidence-Gated RCA — Prove, Then Narrate — Evidence-gated RCA for AI SRE agents: prove with receipts, then narrate. Fixed stages, structural evals, and token-aware tool loops.

• AI Incident Triage for SREs — What Actually Helps On-Call — AI incident triage for SREs — what actually helps on-call versus demo theater. Canonical copy now on StackGen.

• Why One JSON Repair Pass Isn’t Enough for Production Agent Tool Calls — Production agent tool calls need layered JSON repair — why one pass fails on malformed LLM output and what we learned in Go middleware.

• Contributing Back While Building a Commercial Product — Contributing back while building a commercial product: how we shipped a proprietary platform and still merged PRs into the agent framework we depend on.

• AI Agent Runtime vs Platform — Why We Split Them — AI agent runtime vs platform: why we split the Go runtime from Aiden, a multi-tenant orchestration layer for enterprise GenAI agents.

• Terraform for Agent Configuration — Infrastructure as Code Meets AI Governance — Terraform for AI agent configuration — why we use infrastructure as code, not YAML dashboards, to govern production agents.

• You Can’t Debug What You Can’t See — Observability for AI Agents — Observability for AI agents: session traces, tool attribution, token budgets, and audit trails — the signals traditional APM misses in production.

• Your Agent Has Root — Defense-in-Depth for AI Agents That Wield Real Tools — Defense-in-depth for production AI agents: layered policy, HITL, and tool governance when the agent has root — prompts are not security.

• The HITL Paradox — When Human Approval Makes Agents Worse — HITL approvals can make AI agents worse. How to find the human-in-the-loop balance so review gates protect production without stalling the agent.

• Agent Skill Distillation Without Fine-Tuning — Agent skill distillation without fine-tuning: teach reusable skills from production traces instead of GPU-trained distilled models.

• Pensieve — Memory Management for AI Agents That Actually Forget — AI agent memory that forgets on purpose — Pensieve manages four memory types with decay and self-pruning so RAG stops stuffing stale context.

• Implementing ReAcTree — 6 Production Bugs the Paper Didn’t Warn You About — How to debug a multi-step AI agent after you implement ReAcTree: six production bugs the paper skipped, and what actually failed in the loop.

• Go Platform Architecture at Speed — Without Drowning — Go platform architecture for a production AI agent codebase — patterns that keep rapid development sustainable without drowning in process.

• TOML Over YAML and PKL — How We Stopped Fighting Config and Started Shipping — TOML vs YAML vs PKL for agent configuration: why we stopped fighting YAML, skipped PKL, and shipped on TOML.

• Python vs Go for AI Agents — Why We Chose Go — Go vs Python for AI agents: why we chose Go for a production agent runtime — concurrency, single-binary ops, deployment, and when Python still wins.

• LLM Performance Metrics — From Lighthouse to the Token Era — LLM performance metrics for agents: TTFT, time per output token (TPOT), and the token-era equivalents of web vitals — what to measure when Lighthouse no longer applies.

← All tags