# Production Notes > Practitioner notes on production AI agents: evals, guardrails, SRE investigation, and Go runtimes. By Sabith K S, Principal Engineer at StackGen. Prefer Start here and Featured arcs first. Topics, Recent posts, and All posts follow. ## Start here - [Agent evals starter pack](https://productionnotes.dev/start/agent-evals/): Testing pyramid, separate grader, and a supervised E2E learning loop humans close with deploy and re-assert. - [AI agent guardrails with OPA starter pack](https://productionnotes.dev/start/opa-policies/): Authorize tool calls with Open Policy Agent, manage policies as code, time windows, and human approval. - [SRE as service starter pack](https://productionnotes.dev/start/sre-as-service/): Five posts on AI investigation as a service product for on-call, not an engineering credibility project. - [SRE on-call starter pack](https://productionnotes.dev/start/sre-on-call/): Five posts on triage, RCA, and observability for on-call. - [Go agent runtime starter pack](https://productionnotes.dev/start/go-runtime/): Five posts on runtime definition, Go vs Python, and platform split. - [Building an Enterprise AI Agent Platform in Go](https://productionnotes.dev/series/enterprise-ai-agents-go/): Practitioner series on production AI agents in Go: runtime design, workflows, SRE triage, and platform lessons. - [Service Rendered Efficiently](https://productionnotes.dev/series/service-rendered-efficiently/): Sibling series on service culture, investigation product decisions, and operator outcomes. - [What Is an AI Agent Runtime?](https://productionnotes.dev/blog/what-is-an-ai-agent-runtime/): Plain definition of the production agent loop vs platform, chatbot, or notebook demo. - [What Are SRE AI Agents?](https://productionnotes.dev/blog/what-are-sre-ai-agents/): SRE AI agents for triage, RCA, and bounded autonomy, not demo theater. - [Go vs Python for AI Agents: Concurrency, Deployment & Tools](https://productionnotes.dev/blog/why-go/): Go vs Python for production agent runtimes: concurrency, deployment, and when Python still wins. ## Featured arcs - [Agent evals and supervised E2E](https://productionnotes.dev/start/agent-evals/): What belongs in CI, why the grader stays separate, and how coverage grows with human-gated merge, deploy, and re-assert. - [AI agent guardrails with OPA](https://productionnotes.dev/start/opa-policies/): Policy engine outside the model: allow, deny, ask, Terraform attach, time rules. - [Service Rendered Efficiently](https://productionnotes.dev/series/service-rendered-efficiently/): Treat AI investigation as a service product for the teams you support. ## Checklists - [Multi-step AI agent debugging checklist](https://productionnotes.dev/checklists/agent-debug/): Freeze the task, inspect traces and tool evidence, change one host behavior, and replay. - [Evidence-gated RCA checklist](https://productionnotes.dev/checklists/evidence-gated-rca/): Prove with receipts before narrating. - [“Is the agent task done?” checklist](https://productionnotes.dev/checklists/agent-done/): Completion checks that do not self-grade. - [SRE as service checklist](https://productionnotes.dev/checklists/sre-as-service/): Ten yes/no questions for service-shaped AI investigation. ## Topics - [AI Agent Runtime Guides, Debugging & Go Architecture](https://productionnotes.dev/topics/ai-agent-runtime/): A reading map for AI agent runtimes: definitions, Go architecture, debugging, evaluation, observability, and production testing guides. - [AI Agent Workflows](https://productionnotes.dev/topics/ai-agent-workflows/): Debug and verify multi-stage AI agent workflows: stage bring-up, evidence gates, evals, and a supervised E2E learning loop that grows coverage. - [AI Agents for SRE](https://productionnotes.dev/topics/ai-agents-sre/): SRE AI agents for incident triage, RCA, and on-call: what helps versus demo theater, plus observability, evals, and evidence gates. - [AI Incident Triage](https://productionnotes.dev/topics/ai-incident-triage/): AI incident triage for SREs: gather context, form hypotheses, and judge trustworthy RCA without demo theater. Start with the hypothesis ladder. - [ClickHouse on AWS](https://productionnotes.dev/topics/clickhouse-aws/): Self-managed ClickHouse on AWS from first principles: MergeTree internals, engine choices, cluster topology, S3 tiered storage, spot readers, operations, and cost engineering. - [Go AI Agents](https://productionnotes.dev/topics/go-ai-agents/): Go vs Python for AI agents, plus Go platform architecture and why we split the runtime from multi-tenant orchestration. - [Service Rendered Efficiently](https://productionnotes.dev/topics/service-rendered-efficiently/): SRE AI as Service Rendered Efficiently — culture, investigation product decisions, and operator outcomes for on-call teams. ## Recent posts - [When df Lies: keep_free_space, Effective Disk Headroom, and a Fleet That Grows Its Own Volumes](https://productionnotes.dev/blog/clickhouse-effective-disk-headroom-self-healing/): A ClickHouse node showed 30% free in df while merges failed on NOT_ENOUGH_SPACE — the keep_free_space reservation makes df lie by exactly the reservation size. The disk-autoscale design that grows EBS volumes and ext4 online before parts explode. - [The Self-Terminating Alarm: When Per-Host Scale-In Kills Every Scale-Out](https://productionnotes.dev/blog/asg-per-host-scale-in-wedge/): A per-host CloudWatch alarm wired to ASG scale-in terminates its own metric source, stays ALARM forever, and re-kills every scale-out after each cooldown. The anatomy of an autoscaling wedge, and the aggregate-alarm design that fixes it. - [Three Days of Keeper Silence: How a Lost Metric Push Made a Healthy Cluster Blind](https://productionnotes.dev/blog/clickhouse-keeper-silence-alarm-design/): A ClickHouse Keeper monitor sat INSUFFICIENT_DATA for three days — not because the keeper was down, but because a bash-to-Go port silently dropped one line. The alarm design lesson: silence is not health. - [Draining 29,000 S3Queue Files Fast: Every Setting That Matters (With Source Code)](https://productionnotes.dev/blog/clickhouse-s3queue-backlog-drain-optimizations/): Why an S3Queue backlog took 11 hours to drain, the ClickHouse settings that govern every stage — each backed by a permalink into the source — and the exact tuning that took our drain rate up ~8x. - [CAS, SharedMergeTree, or Plain Replication: How to Actually Decide](https://productionnotes.dev/blog/clickhouse-cas-vs-shared-mergetree-decision/): A decision framework for shared storage in ClickHouse: ReplicatedMergeTree on S3, Altinity Antalya CAS, or ClickHouse Cloud SharedMergeTree — with the costs, failure modes, and exit costs of each. - [What Breaks with ClickHouse CAS: Failure Modes, GC, and the Blast Radius](https://productionnotes.dev/blog/clickhouse-cas-failure-modes/): The failure modes that matter for CAS on a real cluster: mount refusal, identity reuse, phantom hosts breaking ON CLUSTER DDL, GC wedges, versioning footguns, and what to alert on. - [Touching CAS Without Betting the Cluster: A Hands-On Staging Walkthrough](https://productionnotes.dev/blog/clickhouse-cas-on-aws-walkthrough/): A staged, hands-on path from per-replica S3 prefixes to a CAS cold tier on AWS: bucket preconditions, the Antalya build, the disk XML, MOVE PARTITION, and the verification queries that tell you it is actually working. - [Content-Addressed Storage in ClickHouse: The Object Model That Replaces Zero-Copy](https://productionnotes.dev/blog/clickhouse-cas-object-model/): How Altinity Antalya's CAS actually works: blobs keyed by hash, manifests, ref logs, mount leases, and an 18-phase GC — and why Keeper is not in the data path. - [ClickHouse zero-copy replication is dead: what actually replaced it](https://productionnotes.dev/blog/clickhouse-zero-copy-replication-dead/): Why upstream ClickHouse deprecated zero-copy replication (PR #82250), what the garbage-collection failure actually was, and what replaced it: ReplicatedMergeTree re-fetches bytes, and CAS shares the pool instead. - [Adopting the Pattern: A Migration Guide for Declarative Node Bootstrap and Lease Identity](https://productionnotes.dev/blog/chinit-adopt-pattern-fleet/): The arc closes with the practitioner question: how would another team take this from posts to their own fleet? A staged migration guide — inventory the boot surface, extract the spec, build the pipeline engine, port the operations in-process, then lease the identity — with the ordering that worked, the traps that cost us live cutovers, and what not to port. For anyone running stateful replicas on a scaling group, on any cloud. - [Who Decides What a Node Is Called? Lease Identity for an Ephemeral Stateful Fleet](https://productionnotes.dev/blog/chinit-lease-identity-ephemeral-fleet/): Stateful replicas in a horizontal scaling group are a contradiction: the group replaces instances freely, but replication paths, Keeper registrations, and per-replica data prefixes key on names that must outlive the instance. This is the lease-based identity pattern that resolves it — a conditional write, a TTL, a renewal heartbeat, and a drain protocol — and the four production findings that shaped each part. Written so you can apply it to any replicated fleet, on any cloud. - [Dissolving the Second Binary: How chinit Absorbed Its Own Companion CLI](https://productionnotes.dev/blog/chinit-dissolve-second-binary/): chinit started life beside a companion CLI that ran the database operator's domain code — two binaries, one process boundary, argv plumbing, and a release pipeline each. This is the story of dissolving that second binary into chinit's in-process operations: the typed request seam that replaced argv, the release-tag lesson that broke our gates twice, and why one binary per node is the right end state. The pattern generalizes to any fleet tooling. ## All posts - [When df Lies: keep_free_space, Effective Disk Headroom, and a Fleet That Grows Its Own Volumes](https://productionnotes.dev/blog/clickhouse-effective-disk-headroom-self-healing/): A ClickHouse node showed 30% free in df while merges failed on NOT_ENOUGH_SPACE — the keep_free_space reservation makes df lie by exactly the reservation size. The disk-autoscale design that grows EBS volumes and ext4 online before parts explode. - [The Self-Terminating Alarm: When Per-Host Scale-In Kills Every Scale-Out](https://productionnotes.dev/blog/asg-per-host-scale-in-wedge/): A per-host CloudWatch alarm wired to ASG scale-in terminates its own metric source, stays ALARM forever, and re-kills every scale-out after each cooldown. The anatomy of an autoscaling wedge, and the aggregate-alarm design that fixes it. - [Three Days of Keeper Silence: How a Lost Metric Push Made a Healthy Cluster Blind](https://productionnotes.dev/blog/clickhouse-keeper-silence-alarm-design/): A ClickHouse Keeper monitor sat INSUFFICIENT_DATA for three days — not because the keeper was down, but because a bash-to-Go port silently dropped one line. The alarm design lesson: silence is not health. - [Draining 29,000 S3Queue Files Fast: Every Setting That Matters (With Source Code)](https://productionnotes.dev/blog/clickhouse-s3queue-backlog-drain-optimizations/): Why an S3Queue backlog took 11 hours to drain, the ClickHouse settings that govern every stage — each backed by a permalink into the source — and the exact tuning that took our drain rate up ~8x. - [CAS, SharedMergeTree, or Plain Replication: How to Actually Decide](https://productionnotes.dev/blog/clickhouse-cas-vs-shared-mergetree-decision/): A decision framework for shared storage in ClickHouse: ReplicatedMergeTree on S3, Altinity Antalya CAS, or ClickHouse Cloud SharedMergeTree — with the costs, failure modes, and exit costs of each. - [What Breaks with ClickHouse CAS: Failure Modes, GC, and the Blast Radius](https://productionnotes.dev/blog/clickhouse-cas-failure-modes/): The failure modes that matter for CAS on a real cluster: mount refusal, identity reuse, phantom hosts breaking ON CLUSTER DDL, GC wedges, versioning footguns, and what to alert on. - [Touching CAS Without Betting the Cluster: A Hands-On Staging Walkthrough](https://productionnotes.dev/blog/clickhouse-cas-on-aws-walkthrough/): A staged, hands-on path from per-replica S3 prefixes to a CAS cold tier on AWS: bucket preconditions, the Antalya build, the disk XML, MOVE PARTITION, and the verification queries that tell you it is actually working. - [Content-Addressed Storage in ClickHouse: The Object Model That Replaces Zero-Copy](https://productionnotes.dev/blog/clickhouse-cas-object-model/): How Altinity Antalya's CAS actually works: blobs keyed by hash, manifests, ref logs, mount leases, and an 18-phase GC — and why Keeper is not in the data path. - [ClickHouse zero-copy replication is dead: what actually replaced it](https://productionnotes.dev/blog/clickhouse-zero-copy-replication-dead/): Why upstream ClickHouse deprecated zero-copy replication (PR #82250), what the garbage-collection failure actually was, and what replaced it: ReplicatedMergeTree re-fetches bytes, and CAS shares the pool instead. - [Adopting the Pattern: A Migration Guide for Declarative Node Bootstrap and Lease Identity](https://productionnotes.dev/blog/chinit-adopt-pattern-fleet/): The arc closes with the practitioner question: how would another team take this from posts to their own fleet? A staged migration guide — inventory the boot surface, extract the spec, build the pipeline engine, port the operations in-process, then lease the identity — with the ordering that worked, the traps that cost us live cutovers, and what not to port. For anyone running stateful replicas on a scaling group, on any cloud. - [Who Decides What a Node Is Called? Lease Identity for an Ephemeral Stateful Fleet](https://productionnotes.dev/blog/chinit-lease-identity-ephemeral-fleet/): Stateful replicas in a horizontal scaling group are a contradiction: the group replaces instances freely, but replication paths, Keeper registrations, and per-replica data prefixes key on names that must outlive the instance. This is the lease-based identity pattern that resolves it — a conditional write, a TTL, a renewal heartbeat, and a drain protocol — and the four production findings that shaped each part. Written so you can apply it to any replicated fleet, on any cloud. - [Dissolving the Second Binary: How chinit Absorbed Its Own Companion CLI](https://productionnotes.dev/blog/chinit-dissolve-second-binary/): chinit started life beside a companion CLI that ran the database operator's domain code — two binaries, one process boundary, argv plumbing, and a release pipeline each. This is the story of dissolving that second binary into chinit's in-process operations: the typed request seam that replaced argv, the release-tag lesson that broke our gates twice, and why one binary per node is the right end state. The pattern generalizes to any fleet tooling. - [One Binary to Bootstrap a Node: Why chinit Replaced 3,600 Lines of Bash](https://productionnotes.dev/blog/chinit-one-binary-node-bootstrap/): Every self-managed database fleet has one: the boot script nobody reads, templated by the IaC tool, drifting from reality. This is the story of replacing ours with chinit — a single Go binary that treats node bootstrap as a declarative spec, a state file, and a retry engine — what the design got right, what three days of live cutovers taught us, and how to tell when your fleet is ready for the same move. - [Destroy, Then Create: The Ordering Rule That Keeps Static-IP ClickHouse Replaceable](https://productionnotes.dev/blog/clickhouse-destroy-then-create-static-ips/): A golden-image roll failed twice with InvalidIPAddress.InUse before I understood why: create_before_destroy flags propagate through the dependency graph, and a flag on an unused TLS certificate was flipping my static-IP database nodes to the wrong replacement order. A tour of Terraform's ForcedCBDTransformer, plan-JSON ordering, and why quorum members and exclusive resources demand destroy-then-create. - [A Field Map of ClickHouse Storage: Disks, Object Stores, and Metadata in the Source](https://productionnotes.dev/blog/clickhouse-storage-classification-source/): After the silent key_prefix bug, I stopped guessing and read ClickHouse's disk registry in the source. Four registrations, three choices per disk, and the default [a-z]{3}/[a-z]{29} key template that explains exactly where the bucket-root prefixes came from. Every claim has a permalink. - [Dead Projections and Useless FINAL: Tuning ClickHouse Tenant Databases by Reading the Source](https://productionnotes.dev/blog/clickhouse-tenant-tuning-from-source/): A fresh-install optimization pass on a multi-tenant ClickHouse cluster — a projection that doubled storage but could never run, a FINAL that forced 20-second merge-on-read, and the Keeper path that blocks per-database tenancy. Every fix verified in the 26.8 source before it shipped. - [The S3 Path Bug Nobody Sees: Silent Config, Shattered Tenants, and a CAS-Ready Fix](https://productionnotes.dev/blog/clickhouse-s3-key-prefix-silent-config-bug/): ClickHouse silently ignored a storage config element on every boot, and every tenant's parts landed in one shared bucket root. How I proved it live, why zero-copy deprecation makes per-replica paths mandatory, and the policy shape that makes CAS a drop-in later. - [Teaching an Operator Without Kubernetes: Building chctl From the Altinity Source](https://productionnotes.dev/blog/clickhouse-operate-without-kubernetes/): Our ClickHouse cluster runs on EC2, not Kubernetes. So we took the Altinity operator's Go source apart, kept the parts that never touched a Pod, and turned them into a CLI that renders configs, syncs schemas, and gates keeper restarts. Here's what was reusable, what wasn't, and the five live bugs the rollout caught. - [Boot Sequence as a Contract: How a Self-Managed ClickHouse Node Comes Up (and How It Fails)](https://productionnotes.dev/blog/clickhouse-boot-sequence-contract/): The 18-step boot pipeline behind a scalable self-managed ClickHouse cluster — what each step does, which failures abort and which warn, and the production findings that forced the ordering: identity before render, secrets before render, criticality as a first-class property. - [The Agent Wrote the Code. Least Privilege Is Still Your Job.](https://productionnotes.dev/blog/coding-agent-least-privilege-cce/): Coding agents can add cloud APIs faster than IAM review. Use CCE to inventory calls, detect privilege drift, and enforce least privilege. - [ClickHouse Config Is Not YAML: What One Re-Render Taught Me About Replica Identity](https://productionnotes.dev/blog/clickhouse-replica-identity-config-render/): I replaced hand-written ClickHouse cluster XML with the Altinity operator's own config generator. One shard macro later, every table silently re-keyed itself in Keeper. A tour of the identity machinery most ClickHouse posts skip. - [Why I Draw ClickHouse Data Flows (and Why EC2 Beat Kubernetes for This Cluster)](https://productionnotes.dev/blog/clickhouse-ec2-vs-k8s-mermaid-data-flow/): The three Mermaid diagrams behind a self-managed ClickHouse cluster — network topology, ingest path, query path — and the honest reasons EC2 beat Kubernetes for it. - [After ReplicatedMergeTree: CAS, Iceberg, and When to Just Buy ClickHouse Cloud](https://productionnotes.dev/blog/clickhouse-whats-next-cas-iceberg/): CAS content-addressed storage, Apache Iceberg, and the honest checklist for choosing self-managed ClickHouse over ClickHouse Cloud. - [The Cost Levers of Self-Managed ClickHouse on AWS](https://productionnotes.dev/blog/clickhouse-aws-cost-engineering/): The real monthly numbers for self-managed ClickHouse on AWS — Graviton, spot readers, S3 tiering, and the janitor that stopped a $1,700/month orphaned-volume leak. - [Operating Self-Managed ClickHouse: Golden Images, Bootstrap, and Runbooks](https://productionnotes.dev/blog/clickhouse-operations-runbooks/): Golden AMIs, two-stage bootstrap, rolling config refresh, and the runbook gaps plain-EC2 ClickHouse owes you versus the Kubernetes operator. - [Running ClickHouse Readers on Spot: The Stable-Identity Problem](https://productionnotes.dev/blog/clickhouse-spot-readers-stable-identity/): A DynamoDB ordinal claimer gives spot-launched ClickHouse readers a stable replica name across instance generations. - [ClickHouse on S3: Tiered Storage Policies That Don't Melt](https://productionnotes.dev/blog/clickhouse-s3-tiered-storage/): How I put all ClickHouse table data in S3 with a tiered storage policy, tuned async inserts so small writes don't hit Too many parts, and why the metadata disk is the one EBS volume that matters. - [Writers, Readers, Keepers: Designing a ClickHouse Cluster That Survives Node Loss](https://productionnotes.dev/blog/clickhouse-cluster-topology-keeper/): How I split a ClickHouse cluster into writers, readers, and keepers, why Keeper quorum is the one thing that flips replicated tables read-only, and how static node identity keeps instance replacement boring. - [SharedMergeTree vs ReplicatedMergeTree: Picking a ClickHouse Engine When You Don't Have ClickHouse Cloud](https://productionnotes.dev/blog/clickhouse-cloud-vs-self-managed-engines/): Cloud's SharedMergeTree does not exist in open source: what you get instead with ReplicatedMergeTree, Keeper, and the zero-copy lesson. - [A Junior Engineer's Guide to ClickHouse: The MergeTree Mental Model](https://productionnotes.dev/blog/clickhouse-mergetree-mental-model/): Parts, merges, and the ORDER BY index: how MergeTree actually stores logs, why tiny inserts break it, and how it compares to row storage. - [Guardrails and human approval for AI agent tool calls](https://productionnotes.dev/blog/ai-agent-guardrails-human-approval/): Allow, deny, or ask: OPA HITL for destructive shell, secret writes, and data exfiltration. Test with opa test before you ship. - [Time-based AI agent policies with OPA](https://productionnotes.dev/blog/time-based-ai-agent-policies-opa/): Business hours and freeze-window Rego for AI agent tool permissions using input.timestamp and OPA time builtins. - [Manage AI agent policies as code with Terraform](https://productionnotes.dev/blog/manage-ai-agent-policies-terraform/): Policy as code for AI agent guardrails: create Rego with sg_policy, attach with sg_agent_policy_attachment, review with plan and apply. - [Authorize AI agent tool calls with OPA](https://productionnotes.dev/blog/authorize-ai-agent-tool-calls-opa/): Open Policy Agent guardrails for AI agents: deny by default where it matters, enforce before tool execution, outside the model. Includes a gh secret set deny lab. - [Why Open Policy Agent for AI agent guardrails](https://productionnotes.dev/blog/why-opa-for-ai-agent-guardrails/): How OPA and Rego came about, why policy engines beat prompt rules, how Pydantic AI and other frameworks wire allow/ask/deny yourself, and what Aiden ships pre-equipped. - [E2E Learning Loop Blind Spots: Skip-as-Success and More](https://productionnotes.dev/blog/e2e-learning-loop-blind-spots/): Most open E2E learning loops are bookkeeping failures: skip-as-success, top-N false absence, merge without deploy, and lost proposals. An operator checklist. - [When Is an E2E Finding Actually Closed?](https://productionnotes.dev/blog/closing-e2e-findings/): Close an E2E finding only after scope, human disposition, runnable asserts, real execution, and post-change verification. Track loop health, not PR volume. - [Two E2E Feedback Loops: Coverage Gaps vs Failure Fixes](https://productionnotes.dev/blog/e2e-coverage-and-failure-loops/): A beginner-friendly guide on why your automated testing robot needs two separate jobs: finding missing tests and fixing broken ones. - [Supervised E2E Learning Loop for AI Agents (Not Auto-Merge)](https://productionnotes.dev/blog/supervised-e2e-learning-loop/): A supervised E2E learning loop makes agent suites stronger over time. Automation drafts coverage and fixes; humans merge, deploy, and re-assert. A draft PR is not closure. - [AI Agent Evals in CI/CD: Flakes, Retries, and Missing Results](https://productionnotes.dev/blog/ai-agent-evals-cicd-flakes/): Keep live agent trials deliberate, report missing tasks and failed attempts, and match trial IDs to traces before trusting a CI result. - [Multi-Agent Handoff Testing: Catch Context Loss Before Production](https://productionnotes.dev/blog/multi-agent-handoff-testing-context-loss/): Test the brief passed from a worker to its parent: settled facts, open gaps, size limits, and a clear stop condition. - [How to Test AI Agent Loops Without Overfitting](https://productionnotes.dev/blog/test-ai-agent-loops-with-evidence/): Detect stalled agent loops from outcomes and evidence, not repeated tool names; test the stop rule with stubbed calls. - [Deterministic Checks vs LLM-as-a-Judge for Agent Evals](https://productionnotes.dev/blog/deterministic-checks-vs-llm-judge/): Check verifiable facts with code and use a calibrated model judge for questions that require reading and interpretation. - [The AI Agent Testing Pyramid: What Belongs in CI?](https://productionnotes.dev/blog/ai-agent-testing-pyramid/): Use fast tests for agent bounds, contract checks for facts, and a small number of live trials for whole jobs. - [AI Agent Evaluation: Why the Grader Must Be Separate](https://productionnotes.dev/blog/ai-agent-evaluation-separate-grader/): Separate the agent from its grader, check facts with code, and run the installed build in the trial environment. - [Same Incident, Five Models: Rank Investigators Without Mixing the Exam](https://productionnotes.dev/blog/same-problem-sre-model-bake-off/): Random CI reruns mix model skill with incident difficulty. Freeze one checkout outage, swap only the bound model, and rank on Correct first, then wall time and new input tokens. - [Session Efficiency Should Not Beat Accuracy](https://productionnotes.dev/blog/session-efficiency-should-not-beat-accuracy/): We labeled a score 0-100 and watched a perfect, cheap, fast agent run print 125. Cap budget bonuses at 1.0 so efficiency never grades above how often you were Correct. - [Chat Completions vs Responses API: Latency, Tools & Reasoning](https://productionnotes.dev/blog/chat-completions-vs-responses-api/): Compare OpenAI Chat Completions vs Responses API for AI agents: tools, reasoning tokens, retries, and latency. Debug a slow run from its trace. - [Hybrid Plan Mode: Smart Planner, Generate Digs](https://productionnotes.dev/blog/hybrid-plan-smart-planner-generate-digs/): Plan roots already route to planning; digs default to tool calling. Putting a high-thinking reasoning model on every task makes workers slow. Use a thinking model for the planner and a normal chat model for digs. - [AI Agent Runtime: What to Measure Before You Buy](https://productionnotes.dev/blog/ai-agent-runtime-what-to-measure-before-you-buy/): Buyer checklist for an AI agent runtime: loop, tools, gates, and an eval receipt — with numbers from a real multi-model triage bench. - [How to Evaluate an AI Agent for Root Cause Analysis](https://productionnotes.dev/blog/how-to-evaluate-ai-agent-root-cause-analysis/): Evaluate AI agent root cause analysis with a 13-point evidence scorecard. Catch unsupported claims, reward honest unknowns, then compare latency and cost. - [Reasoning vs Generate Models for Tool-Heavy Agents](https://productionnotes.dev/blog/reasoning-vs-generate-tool-heavy-agents/): Merits and demerits of reasoning vs normal chat models on a live Grafana triage job — elapsed time, tokens, Theory style, and when each earns the bill. - [Best Way to Debug a Multi-Step AI Agent: A Trace-First Checklist](https://productionnotes.dev/blog/best-way-to-debug-multi-step-ai-agent/): Debug a multi-step AI agent with a reproducible prompt, complete trace, step-level evidence checks, and controlled replay. Fix host failures before prompts. - [Relative Efficiency Scores Lie When Absolute Wall Doubles](https://productionnotes.dev/blog/relative-efficiency-scores-lie/): Efficiency normalized to a cohort median can rise while every seat gets slower. Publish absolute wall, tokens, and correctness first. - [Stop Retrying the Same Failed Observability Query](https://productionnotes.dev/blog/stop-retrying-the-same-failed-query/): Host-side gates for typed query failures, unchanged observations, and same-tool fan-out — why prompt-only retries fail on high-thinking models. - [Hierarchical vs Single-Agent on a Dual-Part Observability Job](https://productionnotes.dev/blog/plan-mode-merits-demerits-observability/): When a hierarchical ReAcTree planner earns its tax on alert+logs triage — and when a single-agent ReAct loop is the better default. Real ratios from a six-combo bench. - [Observability Tools Agents Actually Call on Triage](https://productionnotes.dev/blog/observability-tools-agents-actually-call/): From live combo benches: the short Grafana/Loki tool set that showed up in successful closes — and the dead ends that burned turns. - [What Reasoning Models Write When You Ask Them to Triage](https://productionnotes.dev/blog/what-reasoning-models-write-on-triage/): Same alert+logs prompt, three model classes. How their Theories diverge: leak vs measurement artifact, and what that means for operators. - [Six Model×Orchestration Combos on One Alert+Logs Job](https://productionnotes.dev/blog/six-model-mode-combos-alert-logs-bench/): Same dual-part observability prompt across three model classes and single-agent ReAct vs hierarchical ReAcTree. Wall, tokens, cost, correctness, and an efficiency score you can steal. - [What Is ReAcTree? Hierarchical Agent Trees for Long-Horizon Work](https://productionnotes.dev/blog/what-is-reactree/): Plain English: ReAct is one reason→act→observe loop. ReAcTree is a tree of agent nodes with sequence, parallel, and fallback control flow — for SRE-style tool-heavy jobs. - [Cursor paging for spilled agent tool output](https://productionnotes.dev/blog/cursor-paging-spilled-agent-tool-output/): When observability tool results spill to disk, agents should page with opaque cursors, not grep the preview and declare every pod Unavailable. - [Weekly Reflection: Context Engineering Beat the Language Wars](https://productionnotes.dev/blog/weekly-reflection-context-engineering-wins/): Weekly reflection: context engineering over language wars, plus public libraries and servers for write/select/compress/isolate with multi-language or MCP clients. - [Observation Masking Cut Token Cost 43%—Then Lost the RCA](https://productionnotes.dev/blog/host-reclaim-plan-mode-ab-lessons/): Context engineering A/B: observation masking (tool-result clearing) made an AI SRE agent ~43% cheaper—then failed equal-evidence RCA. Scorecard + knobs. - [Simple vs Plan: When to Use Which (AppWorld smoke cohort)](https://productionnotes.dev/blog/simple-vs-plan-when-to-use-which/): Simple vs plan on AppWorld skills smoke: plan scores higher on some tasks, ties on another, and costs more in this three-task sample. Choose the mode that fits the job — both belong in the toolkit. - [Multi-Agent vs Single-Agent: When Planning Beats Reacting (MCP Tool Tax + pass@k)](https://productionnotes.dev/blog/multi-agent-vs-single-agent-mcp-tool-tax-pass-at-k/): Single-agent MCP loaded 453 tools (~17k peak). A planner peaked lower, spent ~2× total tokens, and sometimes won quality. Report pass@1 and pass@3 — that is not pass^k. - [How to Evaluate AI Agents: Ban Clarifying Questions and Zero-Tool "Success"](https://productionnotes.dev/blog/how-to-evaluate-ai-agents-clarifying-questions-zero-tool-calls/): Unattended AI agent evaluation fails when the model asks the user or reports done with zero MCP calls. Put those rules in the harness, not the framework. - [Debug AI Agent Failures: One Zip Per Conversation](https://productionnotes.dev/blog/one-zip-one-conversation/): Support said the AI investigation went wrong. Give one conversation debug zip—not per-execution scavenger hunts—then grade batches into product gates. - [Running AppWorld Locally for Aiden Agent Evals: Docker Compose, MCP, and Things We Wish We Knew Upfront](https://productionnotes.dev/blog/running-appworld-locally-genie-agent-eval/): Run AppWorld locally with the Aiden agent runtime: docker-compose, MCP HTTP wiring, health gates, and lessons from replacing custom shims with the official three-server stack. - [Stop Spawning Duplicate Workers: What a Handoff Gate Changed in Agent Evals](https://productionnotes.dev/blog/stop-duplicate-agent-workers-handoff-gate/): Stop duplicate agent workers: combined handoff and tool-registration fixes improved fairness to 5/5 and reduced measured planner tokens on one AppWorld slice — partial pass% rose while strict TGC remains a separate harness goal. - [AI Agent Eval Failure Modes: Budget, PII Placeholders, and Self-Reported Passes](https://productionnotes.dev/blog/ai-agent-eval-failure-modes/): AI agent eval failure modes on AppWorld: budget stops before evaluate, PII placeholders poison tool calls, 422 wrong methods, and prose PASS vs judge FAIL. - [Agent Orchestration Tax: Tokens, Iterations, and Tool Calls After a Fair Eval](https://productionnotes.dev/blog/agent-orchestration-tax-evals/): Agent orchestration tax after a fair eval: on AppWorld tasks, planner paths cost ~1.6× tokens and ~3× iterations — measure coordination cost separately from benchmark success. - [Fair Agent Evals: Don't Compare Planner vs Single-Agent Until Tools Match](https://productionnotes.dev/blog/fair-agent-evals-before-performance/): Fair agent evals for planner vs single-agent: match domain tool access on workers before you compare tokens, pass rate, or declare a routing winner. - [Why AI SRE Feels Stuck Before the First Tool Call](https://productionnotes.dev/blog/cut-dead-air-before-investigation/): On-call sees 'the bot is stuck' while vault re-checks and MCP catalog re-index burn cold start. Measure time to the first useful tool call. - [From Vibes to Contracts: How We Rebuilt Agent Evals Around an Industry Standard](https://productionnotes.dev/blog/from-vibes-to-contracts-agent-evals/): From vibes to contracts: how we rebuilt agent evals around eval sets, rubrics vs criteria, a grader stack, and pass^k reliability. - [Canary First: Lessons from Black-Box Consistency Evals for Live SRE Investigate](https://productionnotes.dev/blog/canary-first-sre-investigate-consistency-evals/): Canary-first evals for live SRE investigate: check the canary before burning judge tokens, and never treat a draft RCA as done. - [AI SRE Agent Benchmarks: Wall Time, Tool Calls, Tokens, and ReAcTree Tax](https://productionnotes.dev/blog/ai-sre-agent-benchmarks-wall-time-tools-tokens/): AI SRE agent benchmarks: wall time, tool calls, tokens, and ReAcTree tax — fair A/B numbers so you know when orchestration is worth the cost. - [Reasoning Effort Is Not a Free Upgrade for Tool-Heavy Agents](https://productionnotes.dev/blog/reasoning-effort-is-not-a-free-upgrade/): Reasoning effort is not a free upgrade for tool-heavy AI agents. Live SRE A/Bs: blanket high timed out, adaptive low→high dug deeper. - [Claim-Aware Evidence Packing — Don't Accuse Agents of Inventing What You Truncated](https://productionnotes.dev/blog/claim-aware-evidence-packing/): Claim-aware evidence packing: don't accuse AI agents of inventing what you truncated. Why hallucination guards fail when they drop the receipts. - [Single-Agent vs Multi-Agent Orchestration: How to Choose](https://productionnotes.dev/blog/single-agent-vs-multi-agent/): Single-agent vs multi-agent for SRE triage: a fair A/B, what each shape wins at, and a decision framework so you stop defaulting to either. - [Aiden the Easy Way: One Module from Vague Issue to Review PR](https://productionnotes.dev/blog/aiden-the-easy-way/): Aiden the easy way: one OpenTofu module from a vague GitHub issue to a review PR — Specify, Research, Plan, without wiring every resource by hand. - [Aiden the Hard Way: Vague GitHub Issues to Review PRs](https://productionnotes.dev/blog/from-vague-github-issue-to-pr-with-aiden/): Aiden the hard way: turn a vague GitHub issue into a review PR — GitHub integration, agent, workflow, webhook, and status poll wired by hand. - [What Are SRE AI Agents?](https://productionnotes.dev/blog/what-are-sre-ai-agents/): What are SRE AI agents? AI for incident triage, diagnostics, and RCA with bounded autonomy — not a chatbot and not open-ended remediation demos. - [Don't Invent PromQL: Measure the Alert Rule Query First](https://productionnotes.dev/blog/measure-the-firing-expression-first/): Alert title said latency; the rule was ClickHouse. AI SRE agents that invent PromQL from titles ship wrong RCA. Measure the stored query first. - [What Is an AI Agent Runtime? (Not a Chatbot or Platform)](https://productionnotes.dev/blog/what-is-an-ai-agent-runtime/): An AI agent runtime owns the plan→tool→context loop in production. Not a GenAI platform, chatbot, or notebook. Bounds, traces, and mid-run steer live here. - [SRE for Agentic Systems: Why Uptime Isn't Enough Anymore](https://productionnotes.dev/blog/sre-for-agentic-systems/): SRE for agentic systems: uptime isn't enough. Judgment SLOs and how to measure agentic drift when the agent can be 'up' and still wrong. - [PII Redaction for AI Agents — Two Views, One Trace](https://productionnotes.dev/blog/pii-redaction-ai-agents/): PII redaction for AI agents needs two views: a protected model history and authorized operator visibility when debugging tool calls. - [How to Interrupt, Redirect, or Contain an AI Agent Mid-Run](https://productionnotes.dev/blog/steer-ai-agents-mid-run/): Interrupt, redirect, cancel, or contain an AI agent mid-run. Apply feedback at safe boundaries, preserve evidence, and keep tool policy in force. - [Prompt Caching for AI Agents Is an Architecture Problem](https://productionnotes.dev/blog/prompt-caching-ai-agents/): Prompt caching for AI agents is an architecture problem: stable prefixes, early compaction, and references beat copying a turbulent payload. - [AI Agent Loop Detection — Don't Throw Away the Answer](https://productionnotes.dev/blog/ai-agent-loop-detection-salvage/): AI agent loop detection can erase a good answer. Preserve the best evidence-backed result when a stalled run ends — don't throw the work away. - [Empty PromQL ≠ Missing Data: Fix AI SRE Scope Blindness](https://productionnotes.dev/blog/empty-query-not-absent-signal/): A Grafana no_data result applies to one query and time range; check labels, time, data source, and tool failures before reporting missing evidence. - [AI Agent Root Cause Analysis — Evidence Discarded After the Lead](https://productionnotes.dev/blog/evidence-discarded/): An incident agent can collect a useful metric lead and omit it from the final report; check time windows, unrelated alerts, and transcript-to-summary consistency. - [Be Creative. Don't Invent.](https://productionnotes.dev/blog/be-creative-do-not-invent/): When an incident agent reaches an empty result, broaden searches using recorded identifiers instead of guessing rule IDs or measurements. - [Ungrounded Synthesis Must Read as Hypothesis](https://productionnotes.dev/blog/ungrounded-synthesis-as-hypothesis/): If a check finds unsupported details in an AI-written incident analysis, correct the primary message rather than hiding the warning in a note. - [AI Agent Root Cause Analysis — Curiosity Before Confidence](https://productionnotes.dev/blog/curiosity-before-confidence/): For AI-assisted incident analysis, record required checks, return missing fields together, and limit confidence to what the evidence supports. - [Is the Task Actually Done? — Completion Loops for Production Agents](https://productionnotes.dev/blog/is-the-task-actually-done/): Check an AI agent’s deliverable and tool outcomes before reporting a goal as complete; return unverified work explicitly when checking stops. - [When Your AI Agent Scorecard Lies](https://productionnotes.dev/blog/when-agent-observability-lies/): An AI agent scorecard can grade the wrong work; check trace identity, coverage, and evaluations before using its reliability or quality scores. - [AI Agent Hit Max Turns? Deliver Partial RCA, Not Apology](https://productionnotes.dev/blog/deliver-findings-at-the-budget-cap/): When an incident agent reaches a model-call limit, preserve observed findings and label a partial analysis incomplete. - [The Hypothesis Ladder — Ruling Things Out Before You Narrate](https://productionnotes.dev/blog/hypothesis-ladder/): A structured way to investigate service incidents: identify the affected system and onset, test competing explanations, and report uncertainty. - [From Demo to Deploy — Failure Modes with Receipts](https://productionnotes.dev/blog/demo-to-deploy-receipts/): Questions for evaluating AI incident-response pilots: recorded evidence, staged checks, approval policies, and test results beyond a fluent demo. - [The Diary Learning Loop — From Daily Agent Digests to Human-Approved Policy](https://productionnotes.dev/blog/diary-learning-loop/): AI agent learning loop: daily digests become human-approved workflow and policy changes — not a bigger vector store. - [Beyond Confluence Runbooks: Why GitOps Triage Steps Matter in the AI Era](https://productionnotes.dev/blog/beyond-confluence-runbooks/): Beyond Confluence runbooks: why GitOps triage steps matter for AI agents — version-controlled procedures that change with your stack. - [Your RCA Agent Doesn't Need Another Runbook — It Needs a Map](https://productionnotes.dev/blog/agents-need-a-map-not-a-script/): Your RCA agent doesn't need another runbook — it needs a map. Topology, gates, and verify-first navigation beat a forty-page notebook. - [AI Agents Call Truncated Grafana 'No Data'—It's Spill](https://productionnotes.dev/blog/no-data-is-often-truncated-data/): Large tool results get preview-truncated; AI SRE agents invent 'Unavailable.' Spill recovery and COMPLETE/PARTIAL/FAILED fix dishonest RCA. - [How We Debug Multi-Stage AI Agent Workflows](https://productionnotes.dev/blog/bring-up-agent-workflows-like-hardware/): How to debug multi-step and multi-stage AI agent workflows and execution logs — green one stage at a time, score tool effects not transcripts. - [LLM Tokenomics for Production Agents — Context Budgets as an Operating Model](https://productionnotes.dev/blog/maintaining-tokenomics-with-aiden/): LLM tokenomics for production agents: token budget strategies, context compression, and FinOps loops that keep sessions finishing. - [Evidence-Based Verification — Don't Trust Self-Report, Check the System](https://productionnotes.dev/blog/evidence-based-verification/): Evidence-based verification for AI agents: don't trust self-report — pull proof from ArgoCD, Datadog, and systems of record, then let Go own pass/fail. - [Evidence-Gated RCA — Prove, Then Narrate](https://productionnotes.dev/blog/evidence-gated-multiplane-rca/): Evidence-gated RCA for AI SRE agents: prove with receipts, then narrate. Fixed stages, structural evals, and token-aware tool loops. - [You Vibe-Coded the AWS Calls — Do You Know What IAM Permissions You Actually Need?](https://productionnotes.dev/blog/cce-cloud-entitlements/): Vibe-coded AWS calls hide IAM sprawl. How to discover the cloud entitlements your agent tools actually need — least privilege without guessing. - [Why One JSON Repair Pass Isn't Enough for Production Agent Tool Calls](https://productionnotes.dev/blog/json-repair-layers/): Production agent tool calls need layered JSON repair — why one pass fails on malformed LLM output and what we learned in Go middleware. - [When the Operator Asks to Correlate, Make It a Gate](https://productionnotes.dev/blog/correlate-prior-sessions-gate/): Natural-language correlation goals need server-side gates — not hope the LLM remembers to search prior incidents. - [Contributing Back While Building a Commercial Product](https://productionnotes.dev/blog/open-source-ecosystem/): Contributing back while building a commercial product: how we shipped a proprietary platform and still merged PRs into the agent framework we depend on. - [AI Agent Runtime vs Platform — Why We Split Them](https://productionnotes.dev/blog/aiden-platform/): AI agent runtime vs platform: why we split the Go runtime from Aiden, a multi-tenant orchestration layer for enterprise GenAI agents. - [Terraform for Agent Configuration — Infrastructure as Code Meets AI Governance](https://productionnotes.dev/blog/terraform-config/): Terraform for AI agent configuration — why we use infrastructure as code, not YAML dashboards, to govern production agents. - [You Can't Debug What You Can't See — Observability for AI Agents](https://productionnotes.dev/blog/observability/): Observability for AI agents: session traces, tool attribution, token budgets, and audit trails — the signals traditional APM misses in production. - [Your Agent Has Root — Defense-in-Depth for AI Agents That Wield Real Tools](https://productionnotes.dev/blog/defense-in-depth/): Defense-in-depth for production AI agents: layered policy, HITL, and tool governance when the agent has root — prompts are not security. - [Same Alert, Different Verdict: Entry Path Is Context](https://productionnotes.dev/blog/same-alert-different-verdict/): Don't paste the alert in the UI and wonder why Slack gave a different impact score — entry path carries investigation context. - [The HITL Paradox — When Human Approval Makes Agents Worse](https://productionnotes.dev/blog/hitl-paradox/): HITL approvals can make AI agents worse. How to find the human-in-the-loop balance so review gates protect production without stalling the agent. - [Agent Skill Distillation Without Fine-Tuning](https://productionnotes.dev/blog/skill-distillation/): Agent skill distillation without fine-tuning: teach reusable skills from production traces instead of GPU-trained distilled models. - [Pensieve — Memory Management for AI Agents That Actually Forget](https://productionnotes.dev/blog/pensieve-memory/): AI agent memory that forgets on purpose — Pensieve manages four memory types with decay and self-pruning so RAG stops stuffing stale context. - [Implementing ReAcTree — 6 Production Bugs the Paper Didn't Warn You About](https://productionnotes.dev/blog/reactree-bugs/): Six runtime bugs we found implementing ReAcTree: delegated approvals, graph wiring, timeouts, session isolation, recursion and memory. - [Go Platform Architecture at Speed — Without Drowning](https://productionnotes.dev/blog/anatomy-of-a-platform/): Go platform architecture for a production AI agent codebase — patterns that keep rapid development sustainable without drowning in process. - [TOML Over YAML and PKL — How We Stopped Fighting Config and Started Shipping](https://productionnotes.dev/blog/toml-over-yaml/): Why we chose TOML for flat, typed agent configuration after considering YAML, PKL and CUE, and when those alternatives may fit better. - [Go vs Python for AI Agents: Concurrency, Deployment & Tools](https://productionnotes.dev/blog/why-go/): Compare Go vs Python for AI agents: concurrency, deployment, typed tools, ML libraries, and prototyping. See why our runtime uses Go—and when Python wins. - [Slack Is a Triage Board, Not a Log Dump](https://productionnotes.dev/blog/slack-is-a-triage-board/): Make incident replies in Slack scannable: show counts, findings and uncertainty, with links to evidence and searchable Activity. - [Stop Re-Investigating the Same Alert](https://productionnotes.dev/blog/stop-re-investigating-the-same-alert/): Reuse recent incident findings for follow-ups on the same alert, with a clear way to investigate again when evidence changes. - [Service Rendered Efficiently: SRE AI Is Not an Engineering Credibility Project](https://productionnotes.dev/blog/service-rendered-efficiently/): Service Rendered Efficiently: judge AI investigation by whether it helps on-call reuse results, understand uncertainty, and hand off work. - [LLM Performance Metrics — From Lighthouse to the Token Era](https://productionnotes.dev/blog/web-metrics-to-llm-metrics/): LLM and agent latency: first-token wait, output rate, prompt processing, completion time, and validation failures, using web metrics as a debugging analogy. ## Optional - [About Sabith K S](https://productionnotes.dev/about/): Author bio, background, and team credits behind Production Notes. - [Tags](https://productionnotes.dev/tags/): Browse posts by tag. - [RSS feed](https://productionnotes.dev/feed.xml): Full post feed. - [Newsletter](https://sabithks.substack.com/subscribe): Low-volume Substack subscribe. - [Posts index (JSON)](https://productionnotes.dev/assets/posts-index.json): Machine-readable post catalog for agents and WebMCP tools.