# Production Notes > Enterprise AI agent engineering in Go — production runtimes, workflows, and SRE automation. By Sabith K S, Principal Engineer at StackGen. Written by a practitioner shipping agent infrastructure in production. Prefer the Start here and Topics sections first; Recent posts and All posts follow. ## Start here - [SRE on-call starter pack](https://productionnotes.dev/start/sre-on-call/): Five posts on triage, RCA, and observability for on-call. - [Go agent runtime starter pack](https://productionnotes.dev/start/go-runtime/): Five posts on runtime definition, Go vs Python, and platform split. - [Building an Enterprise AI Agent Platform in Go](https://productionnotes.dev/series/enterprise-ai-agents-go/): Practitioner series on production AI agents in Go — runtime design, workflows, SRE triage, and platform lessons. - [What Is an AI Agent Runtime?](https://productionnotes.dev/blog/what-is-an-ai-agent-runtime/): Plain definition of the production agent loop vs platform, chatbot, or notebook demo. - [What Are SRE AI Agents?](https://productionnotes.dev/blog/what-are-sre-ai-agents/): SRE AI agents for triage, RCA, and bounded autonomy — not demo theater. - [Python vs Go for AI Agents — Why We Chose Go](https://productionnotes.dev/blog/why-go/): Go vs Python for production agent runtimes — concurrency, deployment, and when Python still wins. - [How to Debug Multi-Step AI Agent Workflows](https://productionnotes.dev/blog/bring-up-agent-workflows-like-hardware/): Bring up multi-stage agent pipelines one stage at a time against a golden gate. - [AI Incident Triage for SREs — What Actually Helps On-Call](https://stackgen.com/blog/ai-incident-triage-for-sres-what-works-on-call): What helps on-call versus demo theater (canonical copy on StackGen). ## Checklists - [Evidence-gated RCA checklist](https://productionnotes.dev/checklists/evidence-gated-rca/): Prove with receipts before narrating. - [“Is the agent task done?” checklist](https://productionnotes.dev/checklists/agent-done/): Completion checks that do not self-grade. ## Topics - [AI Agent Runtime](https://productionnotes.dev/topics/ai-agent-runtime/): What an AI agent runtime is — the production loop for planning, tools, and context — vs a GenAI platform, and how we run it in Go. - [AI Agent Workflows](https://productionnotes.dev/topics/ai-agent-workflows/): Production multi-stage AI agent workflows — bring-up discipline, evidence-gated orchestration, and verification patterns for enterprise pipelines. - [AI Agents for SRE](https://productionnotes.dev/topics/ai-agents-sre/): SRE AI agents for incident triage, diagnostics, and RCA — what actually helps on-call teams versus demo theater, plus observability and tokenomics. - [AI Incident Triage](https://productionnotes.dev/topics/ai-incident-triage/): AI for incident triage — what helps on-call SRE teams gather context, form hypotheses, and draft RCA without demo theater. - [Go AI Agents](https://productionnotes.dev/topics/go-ai-agents/): Building enterprise AI agent platforms in Go — language trade-offs, rapid platform architecture, and why we split runtime from multi-tenant orchestration. ## Recent posts - [Claim-Aware Evidence Packing — Don't Accuse Agents of Inventing What You Truncated](https://productionnotes.dev/blog/claim-aware-evidence-packing/): Why hallucination guards fail when they truncate evidence, how claim packing works, and the pitfalls of fail-open and brittle overlap. - [Single-Agent vs Multi-Agent Orchestration: How to Choose](https://productionnotes.dev/blog/single-agent-vs-multi-agent/): Single-agent vs multi-agent orchestration for SRE triage — a fair A/B, what each shape is good at, and a decision framework without defaulting to either. - [Aiden the Easy Way: One Module from Vague Issue to Review PR](https://productionnotes.dev/blog/aiden-the-easy-way/): Adopt Aiden board SDLC with one OpenTofu module — Specify → Research → Plan → review PR — without wiring every sg_* resource by hand. - [Aiden the Hard Way: Vague GitHub Issues to Review PRs](https://productionnotes.dev/blog/from-vague-github-issue-to-pr-with-aiden/): Wire Aiden board SDLC the hard way: every sg_* resource from provider and models through GitHub integration, agent, workflow, webhook, and status poll. - [What Are SRE AI Agents?](https://productionnotes.dev/blog/what-are-sre-ai-agents/): What SRE AI agents are — AI for incident triage, diagnostics, and RCA with bounded autonomy — vs chatbots and open-ended remediation demos. - [What Is an AI Agent Runtime?](https://productionnotes.dev/blog/what-is-an-ai-agent-runtime/): What an AI agent runtime is — the production loop for planning, tools, and context — vs a GenAI platform, chatbot, or notebook demo. - [SRE for Agentic Systems: Why Uptime Isn't Enough Anymore](https://productionnotes.dev/blog/sre-for-agentic-systems/): SRE for agentic systems and AI agents. Introducing Judgment SLOs and how to measure agentic drift in production. - [PII Redaction for AI Agents — Two Views, One Trace](https://productionnotes.dev/blog/pii-redaction-ai-agents/): PII redaction for AI agents needs two views: protected model history and authorized operator visibility when debugging tool calls. ## All posts - [Claim-Aware Evidence Packing — Don't Accuse Agents of Inventing What You Truncated](https://productionnotes.dev/blog/claim-aware-evidence-packing/): Why hallucination guards fail when they truncate evidence, how claim packing works, and the pitfalls of fail-open and brittle overlap. - [Single-Agent vs Multi-Agent Orchestration: How to Choose](https://productionnotes.dev/blog/single-agent-vs-multi-agent/): Single-agent vs multi-agent orchestration for SRE triage — a fair A/B, what each shape is good at, and a decision framework without defaulting to either. - [Aiden the Easy Way: One Module from Vague Issue to Review PR](https://productionnotes.dev/blog/aiden-the-easy-way/): Adopt Aiden board SDLC with one OpenTofu module — Specify → Research → Plan → review PR — without wiring every sg_* resource by hand. - [Aiden the Hard Way: Vague GitHub Issues to Review PRs](https://productionnotes.dev/blog/from-vague-github-issue-to-pr-with-aiden/): Wire Aiden board SDLC the hard way: every sg_* resource from provider and models through GitHub integration, agent, workflow, webhook, and status poll. - [What Are SRE AI Agents?](https://productionnotes.dev/blog/what-are-sre-ai-agents/): What SRE AI agents are — AI for incident triage, diagnostics, and RCA with bounded autonomy — vs chatbots and open-ended remediation demos. - [What Is an AI Agent Runtime?](https://productionnotes.dev/blog/what-is-an-ai-agent-runtime/): What an AI agent runtime is — the production loop for planning, tools, and context — vs a GenAI platform, chatbot, or notebook demo. - [SRE for Agentic Systems: Why Uptime Isn't Enough Anymore](https://productionnotes.dev/blog/sre-for-agentic-systems/): SRE for agentic systems and AI agents. Introducing Judgment SLOs and how to measure agentic drift in production. - [PII Redaction for AI Agents — Two Views, One Trace](https://productionnotes.dev/blog/pii-redaction-ai-agents/): PII redaction for AI agents needs two views: protected model history and authorized operator visibility when debugging tool calls. - [How to Steer an AI Agent Mid-Run Without Starting Over](https://productionnotes.dev/blog/steer-ai-agents-mid-run/): How to steer an AI agent mid-run: additive feedback, iteration-boundary delivery, and completion gates so corrections change the active task. - [Prompt Caching for AI Agents Is an Architecture Problem](https://productionnotes.dev/blog/prompt-caching-ai-agents/): Prompt caching for AI agents fails when context is turbulent — stable prefixes, early compaction, isolated helpers, and references beat payload copies. - [AI Agent Loop Detection — Don't Throw Away the Answer](https://productionnotes.dev/blog/ai-agent-loop-detection-salvage/): AI agent loop detection can erase good work — preserve the best evidence-backed answer when a stalled run ends instead of a generic failure. - [AI Agent Root Cause Analysis — Evidence Discarded After the Lead](https://productionnotes.dev/blog/evidence-discarded/): AI agent root cause analysis fails when digs find a lead and discard it — peer noise, wrong fire-time windows, and transcript gates for AI SRE agents. - [Be Creative. Don't Invent.](https://productionnotes.dev/blog/be-creative-do-not-invent/): When an AI SRE agent hits a dead end during incident response, creativity means searching harder — not hallucinating rule IDs, metrics, or a tidy root cause analysis. - [AI Agent Root Cause Analysis — Curiosity Before Confidence](https://productionnotes.dev/blog/curiosity-before-confidence/): AI agent root cause analysis for SRE: soft prompts don't stop bad RCAs — curiosity checklists, hard gates, and batched validation before confident narratives. - [Is the Task Actually Done? — Completion Loops for Production Agents](https://productionnotes.dev/blog/is-the-task-actually-done/): Why production AI agents need an independent completion check — options we rejected, the goal-scoped loop we shipped, and the papers that shaped it. - [When Your AI Agent Scorecard Lies](https://productionnotes.dev/blog/when-agent-observability-lies/): A production lesson in agent observability: measure telemetry quality before trusting reliability, correctness, cost, or latency scores. - [The Hypothesis Ladder — Ruling Things Out Before You Narrate](https://productionnotes.dev/blog/hypothesis-ladder/): Hypothesis-driven debugging for AI SRE root cause analysis — climb identity and onset before deploy theories, keep parallel branches, prove first and narrate last. - [From Demo to Deploy — Failure Modes with Receipts](https://productionnotes.dev/blog/demo-to-deploy-receipts/): Production-ready AI agents need receipts, not fluent demos — evidence gates, bring-up discipline, HITL tiers, and eval checklists for enterprise agent pilots. - [The Diary Learning Loop — From Daily Agent Digests to Human-Approved Policy](https://productionnotes.dev/blog/diary-learning-loop/): AI agent learning loop for production — daily digests become human-approved workflow and policy changes, not bigger vector stores. HITL governance for enterprise agents. - [Beyond Confluence Runbooks: Why GitOps Triage Steps Matter in the AI Era](https://productionnotes.dev/blog/beyond-confluence-runbooks/): Wiki playbooks trained a generation of on-call engineers. AI agents need procedures that change with your stack — and that's an argument for version-controlled runbooks, not against human-friendly docs. - [Your RCA Agent Doesn't Need Another Runbook — It Needs a Map](https://productionnotes.dev/blog/agents-need-a-map-not-a-script/): Runbooks are fine for humans and stable ladders — bad as the agent's only navigation. After months of multi-plane RCA: topology, gates, and verify-first beat another forty-page notebook. - [How to Debug Multi-Step AI Agent Workflows](https://productionnotes.dev/blog/bring-up-agent-workflows-like-hardware/): The best way to debug a multi-step AI agent and execution logs. Bring up multi-stage agent pipelines one stage at a time. - [LLM Tokenomics for Production Agents — Context Budgets as an Operating Model](https://productionnotes.dev/blog/maintaining-tokenomics-with-aiden/): LLM token budget strategies for production agents — context budgets, tool compression, and FinOps loops that keep sessions finishing. - [Evidence-Based Verification — Don't Trust Self-Report, Check the System](https://productionnotes.dev/blog/evidence-based-verification/): Evidence-based verification for AI agents — pull proof from ArgoCD, Datadog, and systems of record; let Go own pass/fail. - [Evidence-Gated RCA — Prove, Then Narrate](https://productionnotes.dev/blog/evidence-gated-multiplane-rca/): Evidence-gated multi-plane RCA — fixed DAG, structural evals, and token-aware tool loops for production agent workflows. - [You Vibe-Coded the AWS Calls — Do You Know What IAM Permissions You Actually Need?](https://productionnotes.dev/blog/cce-cloud-entitlements/): IAM permissions for vibe-coded AWS calls — cloud entitlements, CCE, and least privilege for agent tools. - [Why One JSON Repair Pass Isn't Enough for Production Agent Tool Calls](https://productionnotes.dev/blog/json-repair-layers/): Production AI agent tool calls need layered JSON repair — why one pass fails and what we learned in Go middleware. - [Contributing Back While Building a Commercial Product](https://productionnotes.dev/blog/open-source-ecosystem/): We built a proprietary product. We also merged 17 PRs into the agent framework we depend on. Here's how to navigate that tension. - [AI Agent Runtime vs Platform — Why We Split Them](https://productionnotes.dev/blog/aiden-platform/): AI agent runtime vs multi-tenant platform for GenAI — why we split the Go runtime from enterprise orchestration (Aiden). - [Terraform for Agent Configuration — Infrastructure as Code Meets AI Governance](https://productionnotes.dev/blog/terraform-config/): We use Terraform to configure our AI agents. Not YAML. Not a dashboard. Terraform. Here's why. - [You Can't Debug What You Can't See — Observability for AI Agents](https://productionnotes.dev/blog/observability/): Observability for AI agents in production: session traces, tool attribution, token budgets, audit trails, and signals traditional APM misses. - [Your Agent Has Root — Defense-in-Depth for AI Agents That Wield Real Tools](https://productionnotes.dev/blog/defense-in-depth/): Secure agent workflows — defense-in-depth for AI agents with real tools: policy, HITL, and layered governance in production. - [The HITL Paradox — When Human Approval Makes Agents Worse](https://productionnotes.dev/blog/hitl-paradox/): Human-in-the-loop is supposed to make agents safer. It can also make them useless. Here's how to find the balance. - [Agent Skill Distillation Without Fine-Tuning](https://productionnotes.dev/blog/skill-distillation/): Agent skill distillation without fine-tuning — teach agents from production traces, not distilled models. Reusable skills without GPU fine-tuning. - [Pensieve — Memory Management for AI Agents That Actually Forget](https://productionnotes.dev/blog/pensieve-memory/): Your agent remembers everything. That's a bug, not a feature. Here's how we built a memory system that learns, forgets, and self-prunes. - [Implementing ReAcTree — 6 Production Bugs the Paper Didn't Warn You About](https://productionnotes.dev/blog/reactree-bugs/): What happens when you take an arXiv algorithm to production. The best way to debug a multi-step AI agent and avoid these 6 ReAcTree bugs. - [Go Platform Architecture at Speed — Without Drowning](https://productionnotes.dev/blog/anatomy-of-a-platform/): Go platform architecture for a production AI agent codebase — patterns that kept rapid development sustainable. - [TOML Over YAML and PKL — How We Stopped Fighting Config and Started Shipping](https://productionnotes.dev/blog/toml-over-yaml/): We tried YAML, considered PKL, and landed on TOML for agent configuration. The reason surprised us. - [Python vs Go for AI Agents — Why We Chose Go](https://productionnotes.dev/blog/why-go/): Go vs Python for AI agents (golang vs python for ai) — production agent runtime, concurrency, deployment, and when Python wins for GenAI. - [LLM Performance Metrics — From Lighthouse to the Token Era](https://productionnotes.dev/blog/web-metrics-to-llm-metrics/): LLM performance metrics — TTFT, time per output token (TPOT), and token-era equivalents of web vitals for agent workloads. ## Optional - [About Sabith K S](https://productionnotes.dev/about/): Author bio, background, and team credits behind Production Notes. - [Tags](https://productionnotes.dev/tags/): Browse posts by tag. - [RSS feed](https://productionnotes.dev/feed.xml): Full post feed. - [Newsletter](https://sabithks.substack.com/subscribe): Low-volume Substack subscribe. - [Posts index (JSON)](https://productionnotes.dev/assets/posts-index.json): Machine-readable post catalog for agents and WebMCP tools.