ClickHouse on AWS starter pack
A short path into self-managed ClickHouse on AWS. The posts are in reading order.
The posts
- A Junior Engineer’s Guide to ClickHouse: The MergeTree Mental Model — parts, merges, and the ORDER BY index, and why tiny inserts break everything
- SharedMergeTree vs ReplicatedMergeTree: Picking a ClickHouse Engine When You Don’t Have ClickHouse Cloud — what Cloud gives you for free and what open source gives you instead
- Writers, Readers, Keepers: Designing a ClickHouse Cluster That Survives Node Loss — split the cluster into three roles so any one node can die at 3 a.m.
- ClickHouse on S3: Tiered Storage Policies That Don’t Melt — all bytes in S3, a local cache, and the one EBS volume you cannot lose
- Running ClickHouse Readers on Spot: The Stable-Identity Problem — spot readers at ~70% off, with a DynamoDB claimer that keeps replica names stable
- Operating Self-Managed ClickHouse: Golden Images, Bootstrap, and Runbooks — the boring machinery that turns EC2 boxes into something like a managed service
- The Cost Levers of Self-Managed ClickHouse on AWS — real monthly numbers, the levers that move them, and the $1,700/month volume leak
- After ReplicatedMergeTree: CAS, Iceberg, and When to Just Buy ClickHouse Cloud — content-addressed storage, open table formats, and the honest buy-vs-build checklist
- Why I Draw ClickHouse Data Flows (and Why EC2 Beat Kubernetes for This Cluster) — the three diagrams that name every hop, and why EC2 won
- ClickHouse Config Is Not YAML: What One Re-Render Taught Me About Replica Identity — the macros, Keeper paths, and raft IDs that name a node — and the re-render that re-keyed them
- The S3 Path Bug Nobody Sees: Silent Config, Shattered Tenants, and a CAS-Ready Fix — a config element ClickHouse silently ignored, proven live with an insert and an
aws s3 ls - Operating ClickHouse Without Kubernetes — the operator’s domain code, packaged as a single binary, against EC2
- Boot Sequence as a Contract: How a Self-Managed ClickHouse Node Comes Up (and How It Fails) — the ordered checks and failure boundaries that make bootstrap and restart behavior inspectable
- Dead Projections and Useless FINAL: Tuning ClickHouse Tenant Databases by Reading the Source — a projection that doubled storage but never ran, a FINAL that cost 20 seconds a query, and the
{database}keeper-path macro that makes per-database tenancy work - A Field Map of ClickHouse Storage: Disks, Object Stores, and Metadata in the Source — disk types, object-storage backends, metadata choices, and the key template behind S3 object layout
- Destroy, Then Create: The Ordering Rule That Keeps Static-IP ClickHouse Replaceable — Terraform replacement ordering, exclusive static IPs, volume identity, and Keeper quorum safety
- One Binary to Bootstrap a Node: Why chinit Replaced 3,600 Lines of Bash — the boot script as a state machine, the typed spec that replaces template variables, and how to know your fleet is ready
- Dissolving the Second Binary: How chinit Absorbed Its Own Companion CLI — the process boundary as a liability, the typed port that replaced argv, and the release-tag lesson that broke the gates twice
- Who Decides What a Node Is Called? Lease Identity for an Ephemeral Stateful Fleet — one conditional write, a TTL lease, a drain protocol, and stateless replacement as the repair strategy
- Adopting the Pattern: A Migration Guide for Declarative Node Bootstrap and Lease Identity — the staged order, the traps that cost live cutovers, what not to port, and the five questions that say you are done
- ClickHouse zero-copy replication is dead: the content-addressed design that replaced it — why upstream deprecated it, how the garbage collection broke, and the CAS object model that replaced it: blobs, manifests, ref logs, mount leases, and why Keeper is not on the data path
- Touching CAS Without Betting the Cluster: A Hands-On Staging Walkthrough — bucket preconditions, the Antalya build, the disk XML, per-partition adoption, and the queries that prove it works
- What Breaks with ClickHouse CAS, and How to Decide If You Should Run It — mount refusal, identity drift, GC wedges, the blast radius — and the weighted scorecard for CAS vs SharedMergeTree vs plain replication, with exit costs
- Draining 29,000 S3Queue Files Fast: Every Setting That Matters — why the backlog formed, every governing setting with source permalinks, and the ~8x drain tuning
- Three Days of Keeper Silence: How a Lost Metric Push Made a Healthy Cluster Blind — the timer that never shipped, the metric push that evaporated in a port, and why liveness alarms must treat silence as failure
- The Self-Terminating Alarm: When Per-Host Scale-In Kills Every Scale-Out — the per-host alarm whose action destroys its own metric, the ASG-wedge anatomy, and the aggregate-alarm fix
- When df Lies: keep_free_space and a Fleet That Grows Its Own Volumes — effective disk headroom, the merge-stall cascade, online EBS + ext4 growth, and the disk-autoscale self-healing loop
- The Fleet That Could Not Heal Itself: TLS Storms, Dead DNS, and What ‘Self-Healing’ Actually Requires — the four error waves of a load soak, and the detection-diagnosis-remedy chain that closed each
- Graviton Roulette: The Day NVMe Device Names Broke Our Boot — device-name roulette, the e2label near-miss on root, and the label-first/partition-skip/patient-wait guards
- S3Queue Is a Multi-Node Engine: Stop Bottlenecking Ingest on One Writer — keeper-claimed files, exactly-once across the fleet, and the one-table fix that turned 100% one-node ingest into single-digit fleet CPU
- The Cloud-Neutral Bootstrapper That Wasn’t: Auditing chinit for Azure — the Fatal assumptions hiding in the “shared” pipeline, the checksum with one signer, and the 18-release pin drift as a symptom of an unbootable port
- From an Empty EC2 Box to a Cluster Member: What Bootstrapping ClickHouse on AWS Actually Means — every AWS primitive a node touches between launch and its first served query, and what chinit does at each of its twenty-four steps
- The Roster, the Journal, and the Bookmarks: What Actually Happens When You CREATE a Replicated ClickHouse Table — the plain-English mental model behind every replication debugging story: membership in Keeper, the shared operations log, per-replica bookmarks, macros, and why idempotent DDL makes convergence possible at all
- The Five-Round Brawl to Make Schema Convergence Automatic — a comment-shredding SQL splitter, ghost replica registrations, a Keeper outage dressed as success, a macro that broke the recovery command, and an ingest table with no business on a backup node — the anatomy of teaching a fleet to converge its own schema
- Postgres Was a Filing Cabinet. ClickHouse Is a Warehouse — the zero-to-hero on-ramp for SQL people: one five-year-old’s picture, every MergeTree concept with its Postgres counterpart (b-tree → sparse index, autovacuum → merges, MVCC → eventual consistency), and the two replication designs that end at SharedMergeTree
- The Loading Dock and the Clipboard: What a ClickHouse S3Queue Table Actually Is — why log pipelines stage files in S3, what a queue table does to a file (discover, claim, parse, insert, delete), and the Keeper clipboard that makes the same queue safe across a fleet — the starting point the backlog-drain and fan-out posts assumed
- Free Is a Route, Not a Setting: The S3 Gateway Endpoint at 30 TB/day — the
$0.045/GBNAT Gateway trap, the three silent-bypass paths (route table, cross-region bucket, hardcoded URL), and the plan-time guards that make “free” an invariant - Don’t Ship Tarballs to a Column Store: ClickHouse Compression from S3Queue to MergeTree — the three compression boundaries in a 30 TB/day pipeline, why
.ndjson.zstbeats.tgzand.gz, and why compression alone doesn’t fix S3 object-count costs - Why the Fix Didn’t Fix It: Stopping a 16.8M-Request S3 API Storm in a ClickHouse Fleet — the five interlocking bugs (
TabSeparated\'false-drift drops,64 GiBhotgp3+10 MiBwide parts, EBSoptimizingtimeouts, Nitro NVMe size flips, and unrolled Launch Templates) that kept an$84/dayS3 API storm alive until every loop was closed - Why ClickHouse Looked 30% Busy While Maxed Out: The 125 MB/s EBS gp3 Wall — why 64 background merges stalled at ~30% user CPU (
0%idle,69%%iowait,805IOPS hitting125 MB/s), and why$20/mofor500 MB/sgp3throughput beat adding$344/mowriters by17x - The Math of ClickHouse CAS on AWS: How Content-Addressed S3 Cuts Cold Storage from R× to 1× — closing the silent S3 Gateway Endpoint leaks (
user-data--region, backup region fallback, sibling VPC route tables), the exact $r \times \frac{V}{c}$ cold-storage and cross-AZ equations at 30 TB/day, the four traps of enabling CAS in Terraform/CI, and reaching0.26¢–0.38¢/GBend-to-end COGS - Meet chinit: Running ClickHouse on Plain Cloud VMs Without Missing Kubernetes — the Apache-2.0 open-source release (
v0.2.6), how the numbered-badge lifecycle works, an honest comparison againstclickhouse-operator, and who it helps most - Three Generations of a 30 TB/Day ClickHouse Fleet: Non-CAS vs. CAS, What Broke, and a Blueprint for Self-Managed S3 — naive stock S3 vs. tuned non-CAS
26.8LTS vs. a live 9-replica Altinity Antalya CAS fleet (1.00×cold S3), the unbiased pros and cons (including theCode: 246fresh-EBS crash-loop and unversioned shared-pool blast radius), and a 5-layer blueprint for future installations
Next
- Hub: ClickHouse on AWS
- Full series: ClickHouse on AWS: Self-Managed from First Principles
- Related: SRE on-call starter pack — the log platform this series describes exists to make on-call investigations faster.
Stay in the loop — production notes on AI agents, workflows, and SRE.
Low volume — new posts and curated reading lists. Unsubscribe anytime.
FAQ
I'm new to ClickHouse. Where should I start?
Read in order. Post 1 explains the MergeTree storage model from zero; each later post defines its terms and links the official docs it builds on. No prior ClickHouse or Kubernetes knowledge is assumed.
Do I need Kubernetes to follow this series?
No. The series is self-managed ClickHouse on EC2 with S3 storage. The final post compares EC2 and Kubernetes honestly, with the conditions that would flip the decision.
Are the numbers real?
Yes — steady-state costs, lease timings, cache sizes, and failure stories come from a production log analytics cluster. Your numbers will differ; the levers and failure modes are the transferable part.