<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://productionnotes.dev/feed.xml" rel="self" type="application/atom+xml" /><link href="https://productionnotes.dev/" rel="alternate" type="text/html" /><updated>2026-07-12T00:19:43+00:00</updated><id>https://productionnotes.dev/feed.xml</id><title type="html">Production Notes</title><subtitle>Production engineering notes on enterprise AI agents, workflows, and SRE — by Sabith K S, Staff Engineer at StackGen.</subtitle><author><name>Sabith K S</name></author><entry><title type="html">How to Debug Multi-Stage AI Agent Workflows — Bring Up Like Hardware</title><link href="https://productionnotes.dev/blog/bring-up-agent-workflows-like-hardware/" rel="alternate" type="text/html" title="How to Debug Multi-Stage AI Agent Workflows — Bring Up Like Hardware" /><published>2026-07-10T17:00:00+00:00</published><updated>2026-07-10T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/bring-up-agent-workflows-like-hardware</id><author><name>Sabith K S</name></author><category term="ai-agents" /><category term="workflows" /><category term="evaluation" /><category term="golang" /><category term="sre" /><category term="testing" /><summary type="html"><![CDATA[Debug multi-stage AI agent workflows by bringing up one stage at a time against golden gates — plus why scoring tool calls beats grading transcripts.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-bring-up-workflows.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-bring-up-workflows.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Maintaining Tokenomics with Aiden — Context Budgets as an Operating Model</title><link href="https://productionnotes.dev/blog/maintaining-tokenomics-with-aiden/" rel="alternate" type="text/html" title="Maintaining Tokenomics with Aiden — Context Budgets as an Operating Model" /><published>2026-07-09T17:00:00+00:00</published><updated>2026-07-09T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/maintaining-tokenomics-with-aiden</id><author><name>Sabith K S</name></author><category term="llm" /><category term="finops" /><category term="ai-agents" /><category term="context-window" /><category term="production" /><category term="aiden" /><summary type="html"><![CDATA[LLM tokenomics for production AI agents — context budgets, tool compression, and FinOps loops that keep sessions finishing.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-tokenomics.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-tokenomics.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Prove, Then Narrate — Deterministic Orchestration Over Autonomous Agents</title><link href="https://productionnotes.dev/blog/evidence-gated-multiplane-rca/" rel="alternate" type="text/html" title="Prove, Then Narrate — Deterministic Orchestration Over Autonomous Agents" /><published>2026-07-08T17:00:00+00:00</published><updated>2026-07-08T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/evidence-gated-multiplane-rca</id><author><name>Sabith K S</name></author><category term="ai-agents" /><category term="compound-ai" /><category term="orchestration" /><category term="evaluation" /><category term="sre" /><category term="workflows" /><summary type="html"><![CDATA[Evidence-gated multi-plane RCA — fixed DAG, structural evals, and token-aware tool loops for production agent workflows.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-evidence-rca.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-evidence-rca.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Evidence-Based Verification — Don’t Trust Self-Report, Check the System</title><link href="https://productionnotes.dev/blog/evidence-based-verification/" rel="alternate" type="text/html" title="Evidence-Based Verification — Don’t Trust Self-Report, Check the System" /><published>2026-07-08T17:00:00+00:00</published><updated>2026-07-08T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/evidence-based-verification</id><author><name>Sabith K S</name></author><category term="ai-agents" /><category term="sre" /><category term="verification" /><category term="observability" /><category term="production" /><category term="golang" /><summary type="html"><![CDATA[Evidence-based verification for AI agents — pull proof from ArgoCD, Datadog, and systems of record; let Go own pass/fail.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-evidence-rca.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-evidence-rca.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">AI Incident Triage for SREs — What Actually Helps On-Call</title><link href="https://productionnotes.dev/blog/ai-incident-triage-sre/" rel="alternate" type="text/html" title="AI Incident Triage for SREs — What Actually Helps On-Call" /><published>2026-07-07T17:00:00+00:00</published><updated>2026-07-07T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/ai-incident-triage-sre</id><author><name>Sabith K S</name></author><category term="sre" /><category term="incident-response" /><category term="on-call" /><category term="ai-agents" /><category term="production" /><category term="golang" /><summary type="html"><![CDATA[AI incident triage for SREs — what actually helps on-call versus demo theater, grounded in parallel context gathering in Go.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-incident-triage.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-incident-triage.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">You Vibe-Coded the AWS Calls — Do You Know What IAM Permissions You Actually Need?</title><link href="https://productionnotes.dev/blog/cce-cloud-entitlements/" rel="alternate" type="text/html" title="You Vibe-Coded the AWS Calls — Do You Know What IAM Permissions You Actually Need?" /><published>2026-07-05T18:00:00+00:00</published><updated>2026-07-05T18:00:00+00:00</updated><id>https://productionnotes.dev/blog/cce-cloud-entitlements</id><author><name>Sabith K S</name></author><category term="cce" /><category term="aws" /><category term="iam" /><category term="security" /><category term="devops" /><category term="ai-coding" /><category term="go" /><category term="github-actions" /><summary type="html"><![CDATA[The complete CCE guide — what Code Context Engine solves, how to install and run it (CLI, Docker, GitHub Actions), recipes, lenses, and reading scan output.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-governance.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-governance.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Why One JSON Repair Pass Isn’t Enough for Production Agent Tool Calls</title><link href="https://productionnotes.dev/blog/json-repair-layers/" rel="alternate" type="text/html" title="Why One JSON Repair Pass Isn’t Enough for Production Agent Tool Calls" /><published>2026-07-03T17:00:00+00:00</published><updated>2026-07-03T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/json-repair-layers</id><author><name>Sabith K S</name></author><category term="ai-agents" /><category term="production" /><category term="go" /><category term="reliability" /><category term="tool-calls" /><summary type="html"><![CDATA[Production AI agent tool calls need layered JSON repair — why one pass fails and what we learned in Go middleware.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-why-go.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-why-go.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Contributing Back While Building a Commercial Product</title><link href="https://productionnotes.dev/blog/open-source-ecosystem/" rel="alternate" type="text/html" title="Contributing Back While Building a Commercial Product" /><published>2026-07-01T17:00:00+00:00</published><updated>2026-07-01T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/open-source-ecosystem</id><author><name>Sabith K S</name></author><category term="open-source" /><category term="community" /><category term="ai-agents" /><category term="go" /><category term="engineering" /><summary type="html"><![CDATA[We built a proprietary product. We also merged 17 PRs into the agent framework we depend on. Here's how to navigate that tension.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-platform.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-platform.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Why We Split Our Agent Runtime From Our Platform</title><link href="https://productionnotes.dev/blog/aiden-platform/" rel="alternate" type="text/html" title="Why We Split Our Agent Runtime From Our Platform" /><published>2026-06-30T17:00:00+00:00</published><updated>2026-06-30T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/aiden-platform</id><author><name>Sabith K S</name></author><category term="aiden" /><category term="platform" /><category term="multi-tenant" /><category term="architecture" /><category term="ai-agents" /><category term="stackgen" /><summary type="html"><![CDATA[Why we split the agent runtime from Aiden — multi-tenant enterprise AI agent platform architecture in Go.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-platform.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-platform.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Terraform for Agent Configuration — Infrastructure as Code Meets AI Governance</title><link href="https://productionnotes.dev/blog/terraform-config/" rel="alternate" type="text/html" title="Terraform for Agent Configuration — Infrastructure as Code Meets AI Governance" /><published>2026-06-29T17:00:00+00:00</published><updated>2026-06-29T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/terraform-config</id><author><name>Sabith K S</name></author><category term="terraform" /><category term="iac" /><category term="gitops" /><category term="ai-agents" /><category term="governance" /><summary type="html"><![CDATA[We use Terraform to configure our AI agents. Not YAML. Not a dashboard. Terraform. Here's why.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-iac.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-iac.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">You Can’t Debug What You Can’t See — Observability for AI Agents</title><link href="https://productionnotes.dev/blog/observability/" rel="alternate" type="text/html" title="You Can’t Debug What You Can’t See — Observability for AI Agents" /><published>2026-06-28T17:00:00+00:00</published><updated>2026-06-28T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/observability</id><author><name>Sabith K S</name></author><category term="observability" /><category term="ai-agents" /><category term="langfuse" /><category term="monitoring" /><category term="production" /><summary type="html"><![CDATA[Observability for production AI agents — session traces, tool attribution, and token budgets beyond traditional APM.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-observability.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-observability.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Your Agent Has Root — Defense-in-Depth for AI Agents That Wield Real Tools</title><link href="https://productionnotes.dev/blog/defense-in-depth/" rel="alternate" type="text/html" title="Your Agent Has Root — Defense-in-Depth for AI Agents That Wield Real Tools" /><published>2026-06-27T17:00:00+00:00</published><updated>2026-06-27T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/defense-in-depth</id><author><name>Sabith K S</name></author><category term="security" /><category term="ai-agents" /><category term="hitl" /><category term="governance" /><category term="production" /><summary type="html"><![CDATA[Your agent can run rm -rf /. Your prompt saying 'don't do that' is not security. Here's why one layer is never enough.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-governance.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-governance.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">The HITL Paradox — When Human Approval Makes Agents Worse</title><link href="https://productionnotes.dev/blog/hitl-paradox/" rel="alternate" type="text/html" title="The HITL Paradox — When Human Approval Makes Agents Worse" /><published>2026-06-26T17:00:00+00:00</published><updated>2026-06-26T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/hitl-paradox</id><author><name>Sabith K S</name></author><category term="hitl" /><category term="ai-agents" /><category term="ux" /><category term="governance" /><category term="production" /><summary type="html"><![CDATA[Human-in-the-loop is supposed to make agents safer. It can also make them useless. Here's how to find the balance.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-hitl.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-hitl.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Teaching Agents to Learn Without Fine-Tuning</title><link href="https://productionnotes.dev/blog/skill-distillation/" rel="alternate" type="text/html" title="Teaching Agents to Learn Without Fine-Tuning" /><published>2026-06-25T17:00:00+00:00</published><updated>2026-06-25T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/skill-distillation</id><author><name>Sabith K S</name></author><category term="ai-agents" /><category term="learning" /><category term="skills" /><category term="llm" /><category term="architecture" /><summary type="html"><![CDATA[Post-session skill distillation from agent traces — how we teach agents to write their own runbooks.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-memory.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-memory.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Pensieve — Memory Management for AI Agents That Actually Forget</title><link href="https://productionnotes.dev/blog/pensieve-memory/" rel="alternate" type="text/html" title="Pensieve — Memory Management for AI Agents That Actually Forget" /><published>2026-06-24T17:00:00+00:00</published><updated>2026-06-24T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/pensieve-memory</id><author><name>Sabith K S</name></author><category term="ai-agents" /><category term="memory" /><category term="rag" /><category term="architecture" /><category term="go" /><summary type="html"><![CDATA[Your agent remembers everything. That's a bug, not a feature. Here's how we built a memory system that learns, forgets, and self-prunes.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-memory.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-memory.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Implementing ReAcTree — 6 Production Bugs the Paper Didn’t Warn You About</title><link href="https://productionnotes.dev/blog/reactree-bugs/" rel="alternate" type="text/html" title="Implementing ReAcTree — 6 Production Bugs the Paper Didn’t Warn You About" /><published>2026-06-23T17:00:00+00:00</published><updated>2026-06-23T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/reactree-bugs</id><author><name>Sabith K S</name></author><category term="ai-agents" /><category term="reactree" /><category term="production" /><category term="bugs" /><category term="go" /><summary type="html"><![CDATA[What happens when you take an arXiv algorithm to production. We found 6 bugs that no paper mentions.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-debug.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-debug.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Architecture at Speed Without Drowning</title><link href="https://productionnotes.dev/blog/anatomy-of-a-platform/" rel="alternate" type="text/html" title="Architecture at Speed Without Drowning" /><published>2026-06-22T17:00:00+00:00</published><updated>2026-06-22T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/anatomy-of-a-platform</id><author><name>Sabith K S</name></author><category term="go" /><category term="architecture" /><category term="ddd" /><category term="engineering" /><category term="ai-agents" /><summary type="html"><![CDATA[From a single Hello World commit to a production Go codebase in a few months — the architecture patterns that made rapid development sustainable.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-platform.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-platform.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">TOML Over YAML and PKL — How We Stopped Fighting Config and Started Shipping</title><link href="https://productionnotes.dev/blog/toml-over-yaml/" rel="alternate" type="text/html" title="TOML Over YAML and PKL — How We Stopped Fighting Config and Started Shipping" /><published>2026-06-21T17:00:00+00:00</published><updated>2026-06-21T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/toml-over-yaml</id><author><name>Sabith K S</name></author><category term="config" /><category term="toml" /><category term="yaml" /><category term="devops" /><category term="ai-agents" /><summary type="html"><![CDATA[We tried YAML, considered PKL, and landed on TOML for agent configuration. The reason surprised us.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-iac.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-iac.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry><entry><title type="html">Go vs Python for AI Agents — Why We Chose Go</title><link href="https://productionnotes.dev/blog/why-go/" rel="alternate" type="text/html" title="Go vs Python for AI Agents — Why We Chose Go" /><published>2026-06-20T17:00:00+00:00</published><updated>2026-06-20T17:00:00+00:00</updated><id>https://productionnotes.dev/blog/why-go</id><author><name>Sabith K S</name></author><category term="go" /><category term="ai-agents" /><category term="architecture" /><category term="llm" /><summary type="html"><![CDATA[Go vs Python for production AI agents — concurrency, deployment, and when you shouldn't follow our path.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://productionnotes.dev/assets/images/og-why-go.png" /><media:content medium="image" url="https://productionnotes.dev/assets/images/og-why-go.png" xmlns:media="http://search.yahoo.com/mrss/" /></entry></feed>