ClickHouse on AWS: Self-Managed from First Principles
This series explains the ClickHouse cluster behind a log analytics platform I run—self-managed on AWS, starting from first principles. It’s written for junior engineers: every term is defined at first use, and every claim carries a number or a link to the official docs. It records not just what we chose but what broke—a $1,700/month EBS leak, a spot-identity race, and deprecated zero-copy replication. Numbers come from a production log analytics cluster; what transfers to your setup is usually the measurement, not our answer.
Start with a pack
| Pack | For |
|---|---|
| ClickHouse on AWS starter pack | The full series in reading order with one-line takeaways |
Topic hubs
Dive by theme:
- ClickHouse on AWS — all posts by topic
Your next reads
-
Start here
A Junior Engineer's Guide to ClickHouse: The MergeTree Mental Model
Parts, merges, and the ORDER BY index: how MergeTree actually stores logs, why tiny inserts break it, and h...
-
Then read
SharedMergeTree vs ReplicatedMergeTree: Picking a ClickHouse Engine When You Don't Have ClickHouse Cloud
Cloud's SharedMergeTree does not exist in open source: what you get instead with ReplicatedMergeTree, Keepe...
-
Then read
Writers, Readers, Keepers: Designing a ClickHouse Cluster That Survives Node Loss
How I split a ClickHouse cluster into writers, readers, and keepers, why Keeper quorum is the one thing tha...
Read in order
Start at post 1 or choose a topic. At the end of each article, use Next in series to continue without returning to this page.
45 posts in this series.
-
Parts, merges, and the ORDER BY index: how MergeTree actually stores logs, why tiny inserts break it, and how it compares to row storage.
-
Cloud's SharedMergeTree does not exist in open source: what you get instead with ReplicatedMergeTree, Keeper, and the zero-copy lesson.
-
How I split a ClickHouse cluster into writers, readers, and keepers, why Keeper quorum is the one thing that flips replicated tables read-only, and how static node identity keeps instance replacement boring.
-
How I put all ClickHouse table data in S3 with a tiered storage policy, tuned async inserts so small writes don't hit Too many parts, and why the metadata disk is the one EBS volume that matters.
-
A DynamoDB ordinal claimer gives spot-launched ClickHouse readers a stable replica name across instance generations.
-
Golden AMIs, two-stage bootstrap, rolling config refresh, and the runbook gaps plain-EC2 ClickHouse owes you versus the Kubernetes operator.
-
The real monthly numbers for self-managed ClickHouse on AWS — Graviton, spot readers, S3 tiering, and the janitor that stopped a $1,700/month orphaned-volume leak.
-
CAS content-addressed storage, Apache Iceberg, and the honest checklist for choosing self-managed ClickHouse over ClickHouse Cloud.
-
The three Mermaid diagrams behind a self-managed ClickHouse cluster — network topology, ingest path, query path — and the honest reasons EC2 beat Kubernetes for it.
-
I replaced hand-written ClickHouse cluster XML with the Altinity operator's own config generator. One shard macro later, every table silently re-keyed itself in Keeper. A tour of the identity machinery most ClickHouse posts skip.
-
ClickHouse silently ignored a storage config element on every boot, and every tenant's parts landed in one shared bucket root. How I proved it live, why zero-copy deprecation makes per-replica paths mandatory, and the policy shape that makes CAS a drop-in later.
-
Our ClickHouse cluster runs on EC2, not Kubernetes. So we took the Altinity operator's Go source apart, kept the parts that never touched a Pod, and turned them into a CLI that renders configs, syncs schemas, and gates keeper restarts. Here's what was reusable, what wasn't, and the five live bugs the rollout caught.
-
The 18-step boot pipeline behind a scalable self-managed ClickHouse cluster — what each step does, which failures abort and which warn, and the production findings that forced the ordering: identity before render, secrets before render, criticality as a first-class property.
-
14 Dead Projections and Useless FINAL: Tuning ClickHouse Tenant Databases by Reading the Source Read
A fresh-install optimization pass on a multi-tenant ClickHouse cluster — a projection that doubled storage but could never run, a FINAL that forced 20-second merge-on-read, and the Keeper path that blocks per-database tenancy. Every fix verified in the 26.8 source before it shipped.
-
After the silent key_prefix bug, I stopped guessing and read ClickHouse's disk registry in the source. Four registrations, three choices per disk, and the default [a-z]{3}/[a-z]{29} key template that explains exactly where the bucket-root prefixes came from. Every claim has a permalink.
-
A golden-image roll failed twice with InvalidIPAddress.InUse before I understood why: create_before_destroy flags propagate through the dependency graph, and a flag on an unused TLS certificate was flipping my static-IP database nodes to the wrong replacement order. A tour of Terraform's ForcedCBDTransformer, plan-JSON ordering, and why quorum members and exclusive resources demand destroy-then-create.
-
Every self-managed database fleet has one: the boot script nobody reads, templated by the IaC tool, drifting from reality. This is the story of replacing ours with chinit — a single Go binary that treats node bootstrap as a declarative spec, a state file, and a retry engine — what the design got right, what three days of live cutovers taught us, and how to tell when your fleet is ready for the same move.
-
chinit started life beside a companion CLI that ran the database operator's domain code — two binaries, one process boundary, argv plumbing, and a release pipeline each. This is the story of dissolving that second binary into chinit's in-process operations: the typed request seam that replaced argv, the release-tag lesson that broke our gates twice, and why one binary per node is the right end state. The pattern generalizes to any fleet tooling.
-
Stateful replicas in a horizontal scaling group are a contradiction: the group replaces instances freely, but replication paths, Keeper registrations, and per-replica data prefixes key on names that must outlive the instance. This is the lease-based identity pattern that resolves it — a conditional write, a TTL, a renewal heartbeat, and a drain protocol — and the five production findings that shaped each part. Written so you can apply it to any replicated fleet, on any cloud.
-
The arc closes with the practitioner question: how would another team take this from posts to their own fleet? A staged migration guide — inventory the boot surface, extract the spec, build the pipeline engine, port the operations in-process, then lease the identity — with the ordering that worked, the traps that cost us live cutovers, and what not to port. For anyone running stateful replicas on a scaling group, on any cloud.
-
Why upstream ClickHouse deprecated zero-copy replication (PR #82250 removed its tests after refcount drift deleted live objects), and the object model that replaced it: Altinity Antalya's CAS, with content-hashed blobs, manifests, ref logs, and reachability-based GC.
-
A staged, hands-on path from per-replica S3 prefixes to a CAS cold tier on AWS: bucket preconditions, the Antalya build, the disk XML, MOVE PARTITION, and the verification queries that tell you it is actually working.
-
The CAS failure modes that matter on a real cluster — mount refusal, identity drift, phantom hosts breaking ON CLUSTER DDL, GC wedges — plus the weighted scorecard for choosing between CAS, SharedMergeTree, and plain replication, with costs and exit costs.
-
Why an S3Queue backlog took 11 hours to drain, the ClickHouse settings that govern every stage — each backed by a permalink into the source — and the exact tuning that took our drain rate up ~8x.
-
A ClickHouse Keeper monitor sat INSUFFICIENT_DATA for three days — not because the keeper was down, but because a bash-to-Go port silently dropped one line. The alarm design lesson: silence is not health.
-
A per-host CloudWatch alarm wired to ASG scale-in terminates its own metric source, stays ALARM forever, and re-kills every scale-out after each cooldown. The anatomy of an autoscaling wedge, and the aggregate-alarm design that fixes it.
-
27 When df Lies: keep_free_space, Effective Disk Headroom, and a Fleet That Grows Its Own Volumes Read
A ClickHouse node showed 30% free in df while merges failed on NOT_ENOUGH_SPACE — the keep_free_space reservation makes df lie by exactly the reservation size. The disk-autoscale design that grows EBS volumes and ext4 online before parts explode.
-
Paper estimates said one thing; a month of 30 TB/day load tests said another. The measured cost model for a ClickHouse-on-EC2 log pipeline — compute, queue, and storage priced per GB with TTL as the only real lever — and the seven lessons that turned a $4,600/month blind spot into a line item you can dial.
-
A four-hour load soak started with 2,284 errors an hour and ended with zero. Between them: a loopback TLS storm invisible to every health check, a cluster config that listed dead nodes by design, an internal CPU monitor that disagreed with the autoscaler, and the discovery that 'self-healing' means the failure has to be *visible* first. Every fix, with source permalinks.
-
We resized the fleet to Graviton and the metadata EBS volume stopped being /dev/nvme1n1. A node bootstrapper that relabels disks by device name nearly ran e2label against the root filesystem — stopped only by an accident of partition tables. A tour of Graviton NVMe enumeration, device-by-label resolution, and the three guards a boot-time disk step needs.
-
Our S3Queue drained from exactly one node while four writers idled — because the queue table only existed on one of them. The engine is natively distributed: every node claims different files through Keeper, exactly-once, and each node's commit caps apply per node. The load test that proved the bottleneck, the one-paragraph fix, and the ClickHouse source that makes the fan-out safe.
-
chinit was designed cloud-neutral — a provider factory, capability probes, a conformance suite. Then we pointed it at the Azure port and found four Fatal assumptions baked into the 'shared' pipeline, an 18-release pin drift, and a checksum gate that could never match. The audit that made the bootstrapper actually portable, and the uncomfortable lessons about what 'cloud-neutral' really costs.
-
'Bootstrap a node' hides about twenty jobs behind two words. This is the complete walkthrough: every AWS primitive a ClickHouse node touches between instance launch and its first served query — the golden AMI, the user-data wrapper, the role spec, the lease claim, the disk mounts, the secret resolution, the config render, the recurring jobs — and what the chinit bootstrapper does at each of its twenty-four steps. Written for engineers who want the whole picture in one place, every term defined at first use.
-
Every hard ClickHouse replication bug — 'replica already exists,' nodes booting with no tables, a recovery command that rejects its own path — makes sense the moment you can picture the three things a CREATE actually builds: a membership roster in the coordination service, a shared journal every replica reads from, and a per-replica bookmark. The plain-English mental model that makes the advanced debugging stories readable: what registers where, who fetches what from whom, what macros really are, and why 'apply the same DDL forever' is safe.
-
A fleet of ClickHouse nodes kept booting with no tables, and every fix revealed the next layer: a SQL splitter that shredded comments into syntax errors, a replica-registration conflict that made CREATE refuse by design, a coordination outage that looked like success, a macro that broke the recovery command itself, and an ingest table that had no business existing on one node at all. The five-round story of teaching a bootstrapper to converge a cluster's schema with no human in the loop — and the install-time validation idea that would have prevented three of the rounds.
-
Everything ClickHouse does sounds weird until you know what it's optimized against. A zero-to-hero walk for people who already know SQL — starting from a five-year-old's question, building the MergeTree model piece by piece with Postgres analogies at every step (indexes, partitions, WAL, autovacuum, replicas), then the two replication designs and why ClickHouse Cloud's SharedMergeTree is the logical end of the same ideas. No prior column-store knowledge assumed.
-
Two earlier posts tuned a slow S3Queue backlog and fixed a one-node ingest bottleneck — both assume you already know why a log pipeline stages files in S3 and what a queue table does with them. This is the missing starting point: from the naive INSERT-directly architecture and its three failure modes, through the staging pattern, to one file's full journey through an S3Queue table — discover, claim, parse, insert, mark — with the Keeper clipboard that makes claiming safe across a fleet. Written for SQL people, one five-year-old picture throughout, no prior queue knowledge assumed.
-
The S3 Gateway Endpoint costs $0/hr and $0/GB — and at 30 TB/day it is the difference between a ~$70/month network bill and a ~$2,700–$9,300/month silent tax. But the endpoint being free and the endpoint being USED are two different guarantees, separated by three independent failure modes: a route-table edit, a cross-region bucket, and a hardcoded URL. This is the anatomy of the $0.045/GB trap, the arithmetic that makes it invisible until it isn't, and the three plan-time guards that turn 'free' from a hope into an invariant — with the real Cost Explorer numbers from a ClickHouse log fleet that got the audit before the bill arrived.
-
A 30 TB/day log pipeline touches compression three separate times: raw .log.gz delivery in S3, staged NDJSON files read by ClickHouse's S3Queue engine, and columnar MergeTree blocks on disk. Confusing those three layers leads to the classic mistakes — wrapping single streams in .tgz archives, burning CPU on gzip-9, leaving staged NDJSON uncompressed, or letting wide parts turn 1.97 TB/day of ZSTD data into 16 million S3 PUT requests. Here is what ClickHouse's C++ source actually supports at each boundary, and how to make the whole chain cost-effective.
-
We shipped shared S3Queue keeper paths, hash-ring filtering, and online EBS disk autoscaling — and AWS Cost Explorer still billed $84.05 in a single partial day for 16.8 million Tier-1 S3 requests. Checking the live fleet against the supposed fixes uncovered five interlocking bugs: a TabSeparated quote-escape parser bug that dropped and recreated S3Queue every five minutes, 64 GiB hot disks hitting move_factor=0.1 with 10 MiB wide parts, an EBS WaitForModification state bug that timed out before resize2fs, a Nitro NVMe enumeration swap, and Launch Templates that had never actually rolled.
-
During a 30 TB/day synthetic log soak on AWS, our ClickHouse writers showed 64 concurrent background merges while CPU user utilization sat at 25–35%. Why wasn't ClickHouse using the remaining CPU? mpstat and iostat revealed the illusion: 0.00% idle, 65–69% iowait, and 100% disk utilization pinned at the exact 125 MB/s EBS gp3 default throughput cap — using only ~805 IOPS. Raising gp3 throughput to 500 MB/s and 4,000 IOPS costs $20/month per writer (versus $262–$344/month to add a writer node) and flipped CPU user utilization from 25.6% to 89.4% within 20 seconds online.
-
An S3 Gateway Endpoint makes the network wire between your VPC and S3 free ($0/hr, $0/GB) — once you close every missing route table and region-less CLI call. What it does NOT make free is storing R physical copies of every cold part across R replicas ($0.023/GB-month × R) or paying $5 per million PUTs to write them. Here is the exact cost math for Altinity Antalya's Content-Addressed Storage (CAS) at 30 TB/day (912,000 GB/month), the four engineering traps we hit enabling it in Terraform and CI, and how CAS takes end-to-end log ingestion down to $0.0026/GB (0.26¢/GB) at 7-day retention and $0.0038/GB (0.38¢/GB) at 33-day retention.
-
We open-sourced chinit (Apache-2.0) — a single Go binary that boots, scales, and self-heals ClickHouse on AWS EC2 and Azure VMs without a Kubernetes cluster. Here is how it works in plain English, an honest look at when it beats (and loses to) clickhouse-operator, and who it is built for.
-
We evolved a 30 TB/day self-managed ClickHouse cluster on AWS through three distinct architectures: naive S3-tiered ReplicatedMergeTree, tuned upstream 26.8 LTS without Content-Addressed Storage (CAS), and a source-tuned CAS fleet on Altinity Antalya + chinit v0.2.7. Here is the unbiased side-by-side comparison with live benchmark telemetry, the real pros and cons of CAS (including Code 246 UUID collisions, Code 48 suspended-versioning traps, asymmetrical reader merge storms, and shared-bucket blast radius), and a single-flag blueprint for future self-managed ClickHouse installations.
-
45 100 TB/Day ClickHouse on AWS: Continuous Spot Data Flows, NVMe Cache, and CAS Cold-Tier Cost Read
An interactive D3 walkthrough of a 100 TB/day (3.04 PB/month) self-managed ClickHouse cluster on AWS with On-Demand writer-0 (backup-wake coordinator), Spot writer-1..9 and NVMe-cached Spot readers, NVMe hot-path compaction, and Altinity Antalya CAS — with exact 7-day and 33-day cluster cost math ($0.0015/GB to $0.0028/GB).
No posts match that search. Try a broader term, or .
Standalone engineering notes are listed on the series index.
Stay in the loop — production notes on AI agents, workflows, and SRE.
Low volume — new posts and curated reading lists. Unsubscribe anytime.
FAQ
Who is this ClickHouse series for?
Engineers starting with ClickHouse on AWS — no prior ClickHouse knowledge assumed. Each post defines its terms, uses real numbers from a production log analytics cluster, and links the official docs it builds on.
Do I need ClickHouse Cloud to follow along?
No. The series is self-managed from first principles: ReplicatedMergeTree, ClickHouse Keeper, and S3 storage policies on EC2. Post 2 explains what ClickHouse Cloud's SharedMergeTree gives you and when buying it wins.
Why EC2 instead of Kubernetes for ClickHouse?
For a small stateful fleet, stable identity under spot churn, Keeper raft discipline, S3-tiered storage, and not paying the cluster tax beat the Altinity operator's conveniences. The final post walks the data flows and the honest reversal condition — if you already run a large EKS fleet, the calculus flips.
What is the chinit arc?
A six-post arc culminating in the Apache-2.0 open-source release of chinit (github.com/stackgenhq/chinit): why one Go binary replaced 3,600 lines of bash (declarative spec, retry policies, state file), why its companion CLI dissolved into in-process typed operations, the lease-based identity protocol that lets stateful replicas live in a horizontal scaling group, a staged migration guide, the Azure portability audit, and an unbiased head-to-head comparison of chinit on cloud VMs versus clickhouse-operator on Kubernetes.
What is the CAS arc?
Five posts on zero-copy for S3-backed ClickHouse: why upstream zero-copy replication was deprecated and the CAS object model that replaced it (content-hashed blobs, manifests, ref logs, leases, an 18-phase GC), a hands-on staging walkthrough, the failure modes and blast radius, the exact 30 TB/day cost math, and a side-by-side three-generation comparison of non-CAS vs. source-tuned CAS (`chinit v0.2.3`, exact 36-byte deterministic UUIDs, `Code: 48` / `Code: 246` / `Code: 234` lessons) as a single-flag blueprint for future installations. CAS is experimental — the arc says so throughout.
What is the self-healing arc?
Three posts from a single incident day: a Keeper liveness push lost in a bash-to-Go port that left the keeper-down alarm blind for three days (and the missing-data semantics that hid it), a per-host scale-in alarm that terminated its own metric source and re-killed every scale-out, and the gap between df and ClickHouse's keep_free_space-aware disk view that stalled merges at 1,800+ parts — each with the automated fix: a restored heartbeat push, an aggregate ASG alarm, and a job that grows EBS volumes + ext4 online before parts hit the insert hard-reject.