Series
Find a focused learning path or jump straight to the newest field notes. Every series lists all of its posts and starts with post 1—no manual table or month-by-month hunting required.
Newest production notes
100 TB/Day ClickHouse on AWS: Continuous Spot Data Flows, NVMe Cache, and CAS Cold-Tier Cost
An interactive D3 walkthrough of a 100 TB/day (3.04 PB/month) self-managed ClickHouse cluster on AWS with On-Demand writer-0 (backup-wake coordinat...
Three Generations of a 30 TB/Day ClickHouse Fleet: Non-CAS vs. CAS, What Broke, and a Blueprint for Self-Managed S3
We evolved a 30 TB/day self-managed ClickHouse cluster on AWS through three distinct architectures: naive S3-tiered ReplicatedMergeTree, tuned upst...
Meet chinit: Running ClickHouse on Plain Cloud VMs Without Missing Kubernetes
We open-sourced chinit (Apache-2.0) — a single Go binary that boots, scales, and self-heals ClickHouse on AWS EC2 and Azure VMs without a Kubernete...
Why ClickHouse Looked 30% Busy While Maxed Out: The 125 MB/s EBS gp3 Wall
During a 30 TB/day synthetic log soak on AWS, our ClickHouse writers showed 64 concurrent background merges while CPU user utilization sat at 25–35...
The Math of ClickHouse CAS on AWS: How Content-Addressed S3 Cuts Cold Storage from R× to 1×
An S3 Gateway Endpoint makes the network wire between your VPC and S3 free ($0/hr, $0/GB) — once you close every missing route table and region-les...
Why the Fix Didn't Fix It: Stopping a 16.8M-Request S3 API Storm in a ClickHouse Fleet
We shipped shared S3Queue keeper paths, hash-ring filtering, and online EBS disk autoscaling — and AWS Cost Explorer still billed $84.05 in a singl...
Choose a series
85 posts
Building an Enterprise AI Agent Platform in Go
Enterprise AI agents in Go: a practitioner series on production runtimes, workflows, SRE triage, and platform lessons from Aiden at StackGen.
View all 85 post titles
- Go vs Python for AI Agents: Concurrency, Deployment & Tools
- TOML Over YAML and PKL — How We Stopped Fighting Config and Started Shipping
- Go Platform Architecture at Speed — Without Drowning
- Implementing ReAcTree — 6 Production Bugs the Paper Didn't Warn You About
- Pensieve — Memory Management for AI Agents That Actually Forget
- Agent Skill Distillation Without Fine-Tuning
- The HITL Paradox — When Human Approval Makes Agents Worse
- Your Agent Has Root — Defense-in-Depth for AI Agents That Wield Real Tools
- You Can't Debug What You Can't See — Observability for AI Agents
- Terraform for Agent Configuration — Infrastructure as Code Meets AI Governance
- AI Agent Runtime vs Platform — Why We Split Them
- Contributing Back While Building a Commercial Product
- Why One JSON Repair Pass Isn't Enough for Production Agent Tool Calls
- Evidence-Gated RCA — Prove, Then Narrate
- AI Incident Triage for SREs — What Actually Helps On-Call
- Evidence-Based Verification — Don't Trust Self-Report, Check the System
- LLM Tokenomics for Production Agents — Context Budgets as an Operating Model
- How We Debug Multi-Stage AI Agent Workflows
- Your RCA Agent Doesn't Need Another Runbook — It Needs a Map
- Beyond Confluence Runbooks: Why GitOps Triage Steps Matter in the AI Era
- The Diary Learning Loop — From Daily Agent Digests to Human-Approved Policy
- From Demo to Deploy — Failure Modes with Receipts
- The Hypothesis Ladder — Ruling Things Out Before You Narrate
- When Your AI Agent Scorecard Lies
- Is the Task Actually Done? — Completion Loops for Production Agents
- AI Agent Root Cause Analysis — Curiosity Before Confidence
- Be Creative. Don't Invent.
- AI Agent Root Cause Analysis — Evidence Discarded After the Lead
- AI Agent Loop Detection — Don't Throw Away the Answer
- Prompt Caching for AI Agents Is an Architecture Problem
- How to Interrupt, Redirect, or Contain an AI Agent Mid-Run
- PII Redaction for AI Agents — Two Views, One Trace
- SRE for Agentic Systems: Why Uptime Isn't Enough Anymore
- What Is an AI Agent Runtime? (Not a Chatbot or Platform)
- What Are SRE AI Agents?
- Aiden the Hard Way: Vague GitHub Issues to Review PRs
- Aiden the Easy Way: One Module from Vague Issue to Review PR
- Single-Agent vs Multi-Agent Orchestration: How to Choose
- Claim-Aware Evidence Packing — Don't Accuse Agents of Inventing What You Truncated
- Reasoning Effort Is Not a Free Upgrade for Tool-Heavy Agents
- AI SRE Agent Benchmarks: Wall Time, Tool Calls, Tokens, and ReAcTree Tax
- Canary First: Lessons from Black-Box Consistency Evals for Live SRE Investigate
- From Vibes to Contracts: How We Rebuilt Agent Evals Around an Industry Standard
- Fair Agent Evals: Don't Compare Planner vs Single-Agent Until Tools Match
- Agent Orchestration Tax: Tokens, Iterations, and Tool Calls After a Fair Eval
- AI Agent Eval Failure Modes: Budget, PII Placeholders, and Self-Reported Passes
- Stop Spawning Duplicate Workers: What a Handoff Gate Changed in Agent Evals
- Running AppWorld Locally for Aiden Agent Evals: Docker Compose, MCP, and Things We Wish We Knew Upfront
- How to Evaluate AI Agents: Ban Clarifying Questions and Zero-Tool "Success"
- Multi-Agent vs Single-Agent: When Planning Beats Reacting (MCP Tool Tax + pass@k)
- Simple vs Plan: When to Use Which (AppWorld smoke cohort)
- Observation Masking Cut Token Cost 43%—Then Lost the RCA
- Weekly Reflection: Context Engineering Beat the Language Wars
- Cursor paging for spilled agent tool output
- What Is ReAcTree? Hierarchical Agent Trees for Long-Horizon Work
- Six Model×Orchestration Combos on One Alert+Logs Job
- What Reasoning Models Write When You Ask Them to Triage
- Observability Tools Agents Actually Call on Triage
- Hierarchical vs Single-Agent on a Dual-Part Observability Job
- Stop Retrying the Same Failed Observability Query
- Relative Efficiency Scores Lie When Absolute Wall Doubles
- Best Way to Debug a Multi-Step AI Agent: A Trace-First Checklist
- Reasoning vs Generate Models for Tool-Heavy Agents
- How to Evaluate an AI Agent for Root Cause Analysis
- AI Agent Runtime: What to Measure Before You Buy
- Hybrid Plan Mode: Smart Planner, Generate Digs
- Chat Completions vs Responses API: Latency, Tools & Reasoning
- Session Efficiency Should Not Beat Accuracy
- Same Incident, Five Models: Rank Investigators Without Mixing the Exam
- AI Agent Evaluation: Why the Grader Must Be Separate
- The AI Agent Testing Pyramid: What Belongs in CI?
- Deterministic Checks vs LLM-as-a-Judge for Agent Evals
- How to Test AI Agent Loops Without Overfitting
- Multi-Agent Handoff Testing: Catch Context Loss Before Production
- AI Agent Evals in CI/CD: Flakes, Retries, and Missing Results
- Supervised E2E Learning Loop for AI Agents (Not Auto-Merge)
- Two E2E Feedback Loops: Coverage Gaps vs Failure Fixes
- When Is an E2E Finding Actually Closed?
- E2E Learning Loop Blind Spots: Skip-as-Success and More
- Why Open Policy Agent for AI agent guardrails
- Authorize AI agent tool calls with OPA
- Manage AI agent policies as code with Terraform
- Time-based AI agent policies with OPA
- Guardrails and human approval for AI agent tool calls
- LLM Performance Metrics — From Lighthouse to the Token Era (related note)
45 posts
ClickHouse on AWS: Self-Managed from First Principles
Self-managed ClickHouse on AWS for engineers starting from first principles: MergeTree internals, engine choices, cluster topology, S3 storage, spot readers, operations, and cost — with the numbers and failure stories behind each choice.
View all 45 post titles
- A Junior Engineer's Guide to ClickHouse: The MergeTree Mental Model
- SharedMergeTree vs ReplicatedMergeTree: Picking a ClickHouse Engine When You Don't Have ClickHouse Cloud
- Writers, Readers, Keepers: Designing a ClickHouse Cluster That Survives Node Loss
- ClickHouse on S3: Tiered Storage Policies That Don't Melt
- Running ClickHouse Readers on Spot: The Stable-Identity Problem
- Operating Self-Managed ClickHouse: Golden Images, Bootstrap, and Runbooks
- The Cost Levers of Self-Managed ClickHouse on AWS
- After ReplicatedMergeTree: CAS, Iceberg, and When to Just Buy ClickHouse Cloud
- Why I Draw ClickHouse Data Flows (and Why EC2 Beat Kubernetes for This Cluster)
- ClickHouse Config Is Not YAML: What One Re-Render Taught Me About Replica Identity
- The S3 Path Bug Nobody Sees: Silent Config, Shattered Tenants, and a CAS-Ready Fix
- Teaching an Operator Without Kubernetes: Building chctl From the Altinity Source
- Boot Sequence as a Contract: How a Self-Managed ClickHouse Node Comes Up (and How It Fails)
- Dead Projections and Useless FINAL: Tuning ClickHouse Tenant Databases by Reading the Source
- A Field Map of ClickHouse Storage: Disks, Object Stores, and Metadata in the Source
- Destroy, Then Create: The Ordering Rule That Keeps Static-IP ClickHouse Replaceable
- One Binary to Bootstrap a Node: Why chinit Replaced 3,600 Lines of Bash
- Dissolving the Second Binary: How chinit Absorbed Its Own Companion CLI
- Who Decides What a Node Is Called? Lease Identity for an Ephemeral Stateful Fleet
- Adopting the Pattern: A Migration Guide for Declarative Node Bootstrap and Lease Identity
- ClickHouse zero-copy replication is dead: the content-addressed design that replaced it
- Touching CAS Without Betting the Cluster: A Hands-On Staging Walkthrough
- What Breaks with ClickHouse CAS, and How to Decide If You Should Run It
- Draining 29,000 S3Queue Files Fast: Every Setting That Matters (With Source Code)
- Three Days of Keeper Silence: How a Lost Metric Push Made a Healthy Cluster Blind
- The Self-Terminating Alarm: When Per-Host Scale-In Kills Every Scale-Out
- When df Lies: keep_free_space, Effective Disk Headroom, and a Fleet That Grows Its Own Volumes
- The Honest Cost Model: Load-Testing a Log Stack Until the Bill Tells the Truth
- The Fleet That Could Not Heal Itself: TLS Storms, Dead DNS, and What 'Self-Healing' Actually Requires
- Graviton Roulette: The Day NVMe Device Names Broke Our Boot (and the Disk-Label Guard That Saved Root)
- S3Queue Is a Multi-Node Engine: Stop Bottlenecking Ingest on One Writer
- The Cloud-Neutral Bootstrapper That Wasn't: Auditing chinit for Azure and What It Taught Us About Portability
- From an Empty EC2 Box to a Cluster Member: What Bootstrapping ClickHouse on AWS Actually Means (and What chinit Does at Every Step)
- The Roster, the Journal, and the Bookmarks: What Actually Happens When You CREATE a Replicated ClickHouse Table
- The Five-Round Brawl to Make Schema Convergence Automatic: Bootstrapping ClickHouse Tables That Refuse to Converge
- Postgres Was a Filing Cabinet. ClickHouse Is a Warehouse: From Zero to SharedMergeTree for SQL People
- The Loading Dock and the Clipboard: What a ClickHouse S3Queue Table Actually Is
- Free Is a Route, Not a Setting: The S3 Gateway Endpoint at 30 TB/day
- Don't Ship Tarballs to a Column Store: ClickHouse Compression from S3Queue to MergeTree
- Why the Fix Didn't Fix It: Stopping a 16.8M-Request S3 API Storm in a ClickHouse Fleet
- Why ClickHouse Looked 30% Busy While Maxed Out: The 125 MB/s EBS gp3 Wall
- The Math of ClickHouse CAS on AWS: How Content-Addressed S3 Cuts Cold Storage from R× to 1×
- Meet chinit: Running ClickHouse on Plain Cloud VMs Without Missing Kubernetes
- Three Generations of a 30 TB/Day ClickHouse Fleet: Non-CAS vs. CAS, What Broke, and a Blueprint for Self-Managed S3
- 100 TB/Day ClickHouse on AWS: Continuous Spot Data Flows, NVMe Cache, and CAS Cold-Tier Cost
12 posts
Service Rendered Efficiently
SRE as Service Rendered Efficiently — a series on AI investigation as a service product for on-call teams, not an engineering credibility project.
View all 12 post titles
- Service Rendered Efficiently: SRE AI Is Not an Engineering Credibility Project
- Stop Re-Investigating the Same Alert
- Slack Is a Triage Board, Not a Log Dump
- Same Alert, Different Verdict: Entry Path Is Context
- When the Operator Asks to Correlate, Make It a Gate
- AI Agents Call Truncated Grafana 'No Data'—It's Spill
- AI Agent Hit Max Turns? Deliver Partial RCA, Not Apology
- Ungrounded Synthesis Must Read as Hypothesis
- Empty PromQL ≠ Missing Data: Fix AI SRE Scope Blindness
- Don't Invent PromQL: Measure the Alert Rule Query First
- Why AI SRE Feels Stuck Before the First Tool Call
- Debug AI Agent Failures: One Zip Per Conversation
Standalone production notes
The Agent Wrote the Code. Least Privilege Is Still Your Job.
Coding agents can add cloud APIs faster than IAM review. Use CCE to inventory calls, detect privilege drift, and enforce least privilege.
You Vibe-Coded the AWS Calls — Do You Know What IAM Permissions You Actually Need?
Vibe-coded AWS calls hide IAM sprawl. How to discover the cloud entitlements your agent tools actually need — least privilege without guessing.
Browse across series
- AI agent runtime · AI agent workflows · AI agents for SRE · Go AI agents · AI incident triage
- Service Rendered Efficiently
- ClickHouse on AWS
Stay in the loop — production notes on AI agents, workflows, and SRE.
Low volume — new posts and curated reading lists. Unsubscribe anytime.
FAQ
How many series does this site have?
Three: Building an Enterprise AI Agent Platform in Go, ClickHouse on AWS, and Service Rendered Efficiently. Each series card below shows its current post count and complete reading order.
How do I read a series from beginning to end?
Choose Start with post 1 or open the full reading order. Every article ends with a Next in series link, so you can continue without returning to the archive.
Where do standalone posts live?
Standalone production notes appear on this page below the series. The newest notes section also includes recent posts from every series and standalone topics.