100 TB/Day ClickHouse on AWS: Continuous Spot Data Flows, NVMe Cache, and CAS Cold-Tier Cost
ClickHouse on AWS: Self-Managed from First Principles · Part 45
Static architecture diagrams hide the thing engineers actually need to reason about: continuous motion across roles. At 100 TB/day (3,040,000 GB/month), four coupled roles run simultaneously every second—writer-0 On-Demand + Spot writer-1..9 ingesting and compacting parts on local NVMe hot disks, a Keeper Raft quorum ordering metadata, cold blobs deduplicating into a shared content-addressed S3 pool, and 100% Spot readers serving queries straight off local NVMe SSDs.
While writer-0 stays pinned On-Demand (on_demand_base_capacity = 1) because it triggers nightly backup-wake for backup-0 and coordinates cluster maintenance, writer-1..9 and all readers run on Spot instances and can be reclaimed by EC2 at any given time. Below is an interactive, color-coded D3 simulation grounded in our production-validated AWS architecture (with all internal cluster names, account IDs, and private domains abstracted). Every role streams continuously in parallel—or you can isolate any section, switch between 7-day and 33-day retention costs, and click Reclaim Spot Writer or Reclaim Spot Reader to watch how NLB drain, DynamoDB leases, local NVMe disks, and zero-byte CAS relinks recover in seconds.
Key takeaways
writer-0stays On-Demand (starts nightly backup);writer-1..9& all readers run 100% Spot:writer-0is pinned On-Demand (writer_spot_on_demand_base_capacity >= 1in Terraform) sochinit backup-wake(02:00 UTC), dead-replica GC, and orphan sweeps are never interrupted by Spot preemption, while 9 Spot writers and 6 Spot NVMe readers absorb scale-out at Spot pricing.- Spot
writer-1..9&reader-0..5can be reclaimed at any time without losing data: On a 2-minute EC2 Spot notice, NLB/replicas_statusdrains traffic to surviving nodes (writer-0stays live), already-committed hot parts are safe on peer replicas via:9010, cold parts are durable in1×S3 CAS, and the replacement Spot instance claims its DynamoDB lease, attaches1×CAS in~2.4s, and syncs recent hot deltas. - NVMe instance-store disks accelerate the hot path and eliminate EBS bottlenecks: Bundled
*gdNVMe SSDs deliver multi-GB/s throughput (~2–4×gp3) at$0extra EBS cost—eliminating%iowaitdisk-bandwidth stalls during~1 GiBMergeTree compactions on writers, keeping write-through hot reads off S3, and leaving zero orphan EBS volumes when a Spot node terminates. - 100 TB/day ClickHouse + CAS cost drops to
$0.0015/GB(7d) and$0.0028/GB(33d): Combining On-Demandwriter-0with diversified Graviton*gdSpotwriter-1..9and readers, NVMe hot & cache disks, the free S3 Gateway Endpoint, and1×CAS cold storage saves$44,941/month(~$539k/year) at 33-day retention compared to10×per-replica S3 copies.
Live 100 TB/Day Architecture Simulation
Use the color-coded tabs to inspect each subsystem, toggle 7d / 33d Retention to see the live monthly cost shift, or click Reclaim Spot Writer / Reclaim Spot Reader to trace a 2-minute Spot interruption and zero-byte CAS recovery.
1. Color-Coded Walkthrough: What Happens in Each Role
1. Writers Fleet · On-Demand writer-0 + Spot writer-1..9 & NVMe Hot Path (Orange)
- On-Demand
writer-0+ Spotwriter-1..9(writer_spot_on_demand_base_capacity = 1): The 10-writer fleet runs16-vCPU / 64 GiB *gdGraviton instances behindingest.clickhouse.example:8443(ACM TLS terminated at the NLB).writer-0stays pinned On-Demand in the writers ASGmixed_instances_policybecause it acts as the ordinal-0 coordinator responsible for startingbackup-0(chinit backup-wakeat02:00 UTC), runningdead-replica-gc/orphaned-gc, and driving vertical autoscale steps. Above that 1-node floor,writer-1..9run 100% Spot across 3*gdfamilies (capacity-optimized, reclaimable anytime) — on a 2-min notice,/replicas_statusdrains new inserts in30stowriter-0and surviving Spot writers. - How the NVMe disk assists the hot path: In load tests at high ingest rates, MergeTree background merges (
127–242 KBsequential blocks) saturated standardgp3EBS bandwidth (125 MB/s) and stalled vCPUs at>65% %iowaitlong before hitting IOPS limits. Putting the hot-path buffer and write-throughcas_cacheon the writer’s bundled local NVMe SSD (~950 GBon16-vCPU *gd,$0extra EBS) provides~2–4×gp3sequential bandwidth, absorbing100 TB/dayburst inserts,3synchronous Materialized Views, and~1 GiBwide-part ZSTD compactions with virtually zero%iowaitstalls. - Metadata safety & intra-AZ
:9010sync: Only a small128 GB gp3volume (delete_on_termination = true) holds the parts catalog/metadata ($102/mototal across 10 writers). ZSTD compacts100 TB/dayraw down to6.58 TB/day(15.2×), and committed hot parts replicate immediately between writers and readers over interserver TLS (:9010) inside a single AZ at$0.00/GBso an abrupt Spot reclaim never loses committed blocks.
2. Keeper Quorum, DynamoDB Spot Leases & Backup (Emerald)
- 3-Node On-Demand Keeper quorum (
:9181): Unlike writers and readers, Keepers never run on Spot. Three 2-vCPU / 8 GiB nodes on dedicatedgp3Raft volumes survive a rolling node replacement without losing write quorum. - DynamoDB node claims (
ch-node-claims): When any Spot writer or reader terminates and a replacement boots,chinitexecutes a conditional DynamoDB write to claim an ordinal slot (writer-N/reader-N) and writes a deterministic server UUID so CAS mount ownership never hitsCode: 246conflicts. - Nightly isolated backup (
writer-0→backup-0): On-Demandwriter-0invokeschinit backup-wakeat02:00 UTCto wake the stoppedbackup-0EC2 instance, which syncs metadata, writes tar snapshots to a separate versioned/object-locked backup bucket, and self-stops.
3. CAS Cold Pool & Lifecycle (Purple) — 1× Storage + Zero-Byte Relink
- Free S3 Gateway Endpoint (
$0.00/GB): All ClickHouse-to-S3 traffic rides the regional VPC Gateway Endpoint, avoiding$18,000+/monthin$0.045/GBNAT Gateway data-processing charges. - 1× Shared XXH3-128 Pool: Under
tiered_cas(perform_ttl_move_on_insert = false), merged~1 GiBwide parts age off the NVMe hot path (TTL +1–7dormove_factor = 0.1pressure spill), hash intoXXH3-128blobs, and upload once (1×) into an unversioned S3 CAS bucket (s3://org-ch-cas-cold-pool/cas/) with no{replica}segment in the path. - Zero-byte relink (
CA_RELINK_COOKIE): Replacement Spot writers and Spot readers never copy gigabytes of cold history over:9010. They verifyHEADon S3 and attach the manifest in~2–3 secondswith0 bytestransferred; writers then pull only recent un-aged hot delta parts over:9010. - GC & Avoiding the S3 Standard-IA Early-Deletion Trap: An elected CAS GC leader folds ref-logs every
300s(cas_gc_interval_sec = 300) using S3If-Matchconditional writes. Crucial FinOps fix: for7d–33dretention windows, we setlifecycle_days_to_ia = 0and keep cold parts in S3 Standard ($0.023/GB-mo). Transitioning to S3 Standard-IA at day30when ClickHouse TTL drops parts at day31–33triggers AWS’s 30-day minimum storage duration early-deletion fee—making those final 3 days cost~$2,467/moinstead of~$454/moin S3 Standard (>5×more expensive). Standard-IA only pays off for longer retention (≥ 60–90+days) where objects stay in IA well past the 30-day floor.
4. Readers Fleet · 100% Spot Replicas & NVMe Instance-Store Cache (Sky Blue)
- Read-only Spot fleet behind
/replicas_status: Query clients (query.clickhouse.example:8443) hit an NLB target group gated byHTTP :8123/replicas_status, ensuring newly booted Spot readers finish relinking and hot backfill before serving traffic. - Why NVMe instance store is ideal here (
1.8 GB/svs.131–149 MB/sgp3): Readers run6×4-vCPU / 16 GiB Graviton NVMe Spot instances (*gdpool). With a role-scoped Terraform flag (reader_cache_on_nvme_instance_store = truegated onalltrue*gdpool membership so EBS-only roles never lose theirgp3cache), Terraform drops/dev/sdccompletely ($0cache EBS) andchinitformats and mounts/dev/nvme2n1(217 GiBusable onm8gd.xlarge,4 × 48.65 GiBwrappers =3.4×larger than a64 GiBgp3). Direct-IOddbenchmarks on the live reader measured1.8 GB/ssequential read (287 MB/swrite) vs.131–149 MB/son the baseline64 GiBgp3cache volume (~13×faster). - Answering the architectural question — “Do replicas still pull the same hot parts into their own NVMe cache?”: No. Under
tiered_cas, hot parts and the NVMe cache are distinct layers, and replicas never eagerly pull the shared S3 pool onto NVMe:- Hot parts (
< cold move TTL) live on each replica’s<hot>volume (defaultdisk) and replicate peer-to-peer over:9010— they never entercas_cache. - Cold parts (
> cold move TTL) live once (1.00×) in S3 CAS and attach on peers viaCA_RELINK_COOKIEwith0bytes downloaded (FilesystemCacheBytes = 0after relink; only the single replica that ran the TTL move warms bytes viacache_on_write_operations = 1). - NVMe
cas_cacheis strictly a lazySLRUbyte-range cache for cold S3: a reader only populates4 MiB–32 MiBsegments of the exact cold columns/granules scanned by liveSELECTqueries. Consequently, if your cold-move TTL is7 DAY,cas_cachestays0 Bidle for the first 7 days! Tightening the move rule toTTL timestamp + INTERVAL 1 DAY TO VOLUME 'cold'shrinks per-replica hot storage by~75–85%(cutting Spot reader hot-backfill time from79sto<15s) and puts days2–33directly onto1×S3 CAS +1.8 GB/slocal NVMe cache.
- Hot parts (
2. Live Chaos Validation: 81.86 TB/day Burst Across 100% Spot + NVMe Readers
Before relying on modeled numbers, we parked a live dev/staging cluster on 100% Spot Writers (m8g/m7g/m6g.4xlarge, writer_min = 1) + 100% NVMe Spot Readers (m8gd/m7gd.xlarge + c8gd.2xlarge, reader_cache_on_nvme_instance_store = true) and fired an 81.86 TB/day (466,820 rec/s) 10-minute S3Queue burst (2,925 files, 571.3 GB raw NDJSON = 625.6 GiB uncompressed columnar) right through an active AWS Spot capacity squeeze:
Measured telemetry from our 81.86 TB/day 100% Spot + NVMe reader stress test: 334,739,767 exact rows across all 6/6 Spot replicas (0 data loss), 49s S3 staging drain, 15 Spot launches / 6 real AWS Spot reclaims survived in 100 minutes, and a 46% live run-rate cost drop ($3.26/hr → $1.75/hr).
Four concrete operational findings emerged from that 100-minute Spot churn window:
- Zero data loss across
6AWS Spot reclaims and a mid-burst3 → 2 → 4writer swing: Pre-burst baseline (53,443,013rows) +281,296,754generated rows converged to334,739,767exact rows (744.47 GiBuncompressed →47.81 GiBcompressed =15.57×ZSTD) on all6/6replicas, withstagingdrained to0objects49 secondsafter the generator finished and$0.0662total S3 API cost (0Code: 241OOMs). Even when a pre-burst idle alarm stepped writers down3 → 290 seconds into the burst, 2 Spot writers solo-absorbed81.86 TB/dayat99.6%CPU (0–1slag) until CPU TargetTracking scaled2 → 4. - Single-AZ
*gdNVMe Spot pools are shallow — Multi-AZ + CAS fixes it: Pinningm8gd/m7gd.xlargeSpot readers to a single AZ triggered 5 reader Spot reclaims in 60 minutes (including a3.5mUnfulfillableCapacitypinch). Because CAS cold parts relink across AZs via S3 at$0.00/GB, enablingmulti_az_configacross 3 AZs and addingr8gd.xlarge/r7gd.xlarge/c7gd.2xlargeexpands Spot pool diversity6×(18pools) whilemin_readers >= 2(orreader_spot_ratio = 0.5) eliminates0-readerwindows. HTTP :8123/replicas_statusprevented every cold-reader query blackhole: On every Spot replacement (128schinitboot +79s:9010hot backfill for334.74Mrows),/replicas_statusheld the fresh readerunhealthyon the NLB untillag = 0s, then flipped ithealthyautomatically.- Simultaneous hot backfill proved why writer hot NVMe (
*gd) is critical at 100 TB/day: Steady81.86 TB/dayingest held500 MB/swritergp3at0.8–18.4%%iowait, until 2 scaled-out writers and 2 replacement Spot readers simultaneously fetched48 GiBof hot parts — spiking writergp3%iowaitto45.4%–85.6%. Moving writer hot storage onto*gdlocal NVMe (1.8–3 GB/s) removes that backfill + merge I/O ceiling completely.
3. 100 TB/Day Cost Breakdown (3,040,000 GB/Month, us-west-2)
Using our measured 15.2× ZSTD compression ratio, 100 TB/day produces 6.58 TB/day (6,579 GB/day) of net compressed ClickHouse data:
- 7-day cold pool (
1×CAS):46.1 TB(46,053 GB) in S3 Standard ($0.023/GB-mo) =$1,059/month - 33-day cold pool (
1×CAS):217.1 TB(217,105 GB) kept in S3 Standard ($0.023/GB-mo,lifecycle_days_to_ia = 0) =$4,993/month
| Cost Line Item (100 TB/day = 3.04M GB/mo) | 7-Day Retention | 33-Day Retention | Notes & Cost Levers |
|---|---|---|---|
Writers Fleet (1× OD writer-0 + 9× *gd Spot) |
$2,290/mo |
$2,290/mo |
writer-0 pinned On-Demand (~$580/mo) for backup-wake; writer-1..9 Spot ($1,710/mo); $0 NVMe hot disk |
Writer Metadata EBS (10× 128 GB gp3) |
$102/mo |
$102/mo |
Catalog only (delete_on_termination=true); hot/merge path rides NVMe |
Readers Fleet (6× 4-vCPU / 16 GiB *gd Spot) |
$330/mo |
$330/mo |
100% Spot (*gd NVMe pool); NVMe cas_cache = $0 EBS |
Reader Metadata EBS (6× 64 GB gp3) |
$31/mo |
$31/mo |
Small catalog disk only (cas_cache rides local NVMe) |
Keeper Quorum (3× 2-vCPU / 8 GiB On-Demand) |
$162/mo |
$162/mo |
On-Demand Raft quorum + dedicated 64 GB gp3 fsync disks |
CAS Cold S3 Pool (1× Shared XXH3-128) |
$1,059/mo |
$4,993/mo |
46.1 TB @ 7d · 217.1 TB @ 33d (all S3 Standard; no 30d IA early-deletion fee) |
S3 API Requests (Wide Parts + gc_interval=300s) |
$580/mo |
$580/mo |
~1 GiB wide parts + 300s GC folding keep Tier-1 PUTs low |
NLBs + warning CW Logs + DynamoDB + Backup |
$150/mo |
$150/mo |
Verbose logs stay local in ZSTD system.text_log; S3 Gateway = $0/GB |
Total ClickHouse + CAS Cluster (100 TB/day) |
$4,704/mo |
$8,638/mo |
$0.0015/GB (0.15¢/GB) @ 7d · $0.0028/GB (0.28¢/GB) @ 33d |
What CAS + Spot + NVMe Saves at 100 TB/Day
| Architecture Variant (33-Day Retention, 100 TB/day) | Cold S3 Storage | Total Cluster Cost | Savings vs. Non-CAS 10× |
|---|---|---|---|
Non-CAS Unsharded (10× per-replica S3 copies) |
$49,934/mo |
~$53,579/mo |
Baseline |
Non-CAS Sharded (S = 5 shards × r = 3 = 3× S3) |
$14,980/mo |
~$18,625/mo |
-$34,954/mo (-65%) |
Antalya CAS (1× S3) + OD w0 + Spot + NVMe |
$4,993/mo |
~$8,638/mo |
-$44,941/mo (-84%) |
4. Production Caveats & Operational Trade-offs
Good FinOps engineering makes the trade-offs explicit alongside the savings:
- S3 Standard-IA minimum duration (
30days): Never transition cold parts to Standard-IA unless parts survive well pastlifecycle_days_to_ia + ~17days (for example,60–90+day retention with a day-30 transition). For≤ 47-day TTLs, keeplifecycle_days_to_ia = 0in S3 Standard so early-deletion charges never erase the lower storage rate. - Experimental status of Altinity Antalya CAS: CAS (
metadata_type = cas) is marked experimental in Antalya26.6+/26.8builds. Pin your Golden AMI build version, monitorPartsLockWaitMicrosecondsand CAS GC concurrency (cas_gc_interval_sec = 300), ensure versioning and Object Lock remain disabled on the CAS bucket, and drill restores frombackup-0before promoting CAS to strict-SLA production. - Spot writer floor under regional capacity crunches: Pinning
writer-0On-Demand (writer_spot_on_demand_base_capacity = 1) keepsbackup-wakeand maintenance safe, and spreadingwriter-1..9across 3*gdfamilies minimizes correlated reclaims. However, if a regional Spot crunch evicts multiple writers faster than:9010replication drains, workloads with strict zero-backpressure SLAs should raisewriter_spot_on_demand_base_capacityto theirwriter_miningest quorum floor (or keepwriter_spot_ratio = 0.0). - Single-AZ compute vs. cross-AZ
:9010transfer: Placing writers, readers, and Keepers in one AZ eliminates AWS’s$0.02/GBround-trip cross-AZ data-transfer fee on:9010hot replication ($4,000–$8,000+/mosaved at this ingest scale), while S3 still preserves 3-AZ data durability. The trade-off is AZ-level compute availability (an AZ outage becomes an ASG re-launch into a healthy AZ rather than zero-second failover). - Compression sensitivity (
15.2×vs.3×–6×): The$4,704–$8,638/monumbers assume15.2×ZSTD compression measured on structured observability logs/telemetry. High-entropy payloads (random UUIDs, hashes, unindexed JSON) typically compress at3×–6×, which increases S3 cold storage and:9010/S3 bandwidth by~2.5–5×—though1×CAS still eliminates the10×replica-copy multiplier either way.
At 100 TB/day, pinning writer-0 On-Demand to anchor backups and maintenance while moving hot-path compaction and cold-read caching onto free NVMe instance-store disks, running Spot compute for writer-1..9 and all readers, and deduplicating cold S3 into a 1× CAS pool turns a $53.6k/month storage-bound cluster into a $4.7k–$8.6k/month elastic system under 0.28¢/GB.
Previous: Three Generations of a 30 TB/Day ClickHouse Fleet: Non-CAS vs. CAS, What Broke, and a Blueprint for Self-Managed S3. Series index: ClickHouse on AWS: Self-Managed from First Principles.
FAQ
Why does writer-0 stay On-Demand while writer-1..9 and all readers run on Spot?
writer-0 is the cluster's ordinal-0 maintenance and backup coordinator: it triggers `chinit backup-wake` on schedule (02:00 UTC) to wake the on-demand backup replica (`backup-0`), and runs dead-replica GC, orphan volume sweeps, and autoscale coordination. Pinning `writer_spot_on_demand_base_capacity = 1` in the writers ASG keeps writer-0 On-Demand (never Spot-reclaimed) while letting writer-1..9 and reader-0..5 run 100% Spot on diversified Graviton *gd pools. When EC2 reclaims any Spot writer or reader with a 2-minute notice, NLB /replicas_status drains traffic in 30s, committed hot parts stay safe via :9010 replication, and the replacement claims its DynamoDB lease, formats its NVMe disk, and relinks 1x S3 CAS parts in ~2-3 seconds.
How does the NVMe disk assist the hot path on Spot writers as well as readers?
On 100 TB/day writers, MergeTree background merges and synchronous Materialized Views write large sequential blocks (127–242 KB) that easily saturate standard gp3 EBS bandwidth and stall CPUs at >65% %iowait. Using local NVMe instance-store disks (*gd Spot pools) on the writer hot path delivers multi-GB/s sequential throughput (~2–4x gp3) at $0 extra EBS cost, absorbs burst inserts and ~1 GiB wide-part ZSTD compaction without %iowait stalls, serves write-through hot queries locally, and disappears cleanly with the VM on Spot termination (zero orphan EBS volumes).
What does 100 TB/day (3.04 PB/month) cost with Spot compute, NVMe hot/read cache, and CAS?
At 15.2x ZSTD compression (6.58 TB/day compressed), the ClickHouse + CAS cluster (including On-Demand writer-0 and 3x On-Demand Keepers) costs ~$4,704/month at 7-day retention ($0.0015/GB) and ~$8,638/month at 33-day retention ($0.0028/GB, kept in S3 Standard to avoid Standard-IA's 30-day minimum early-deletion fee). Compared to an unsharded 10-replica non-CAS cluster ($49,934/month in cold S3 alone at 33 days), CAS saves ~$44,941/month (~$539k/year).
Why keep hot parts on local NVMe for writers instead of writing directly to S3?
Setting perform_ttl_move_on_insert = false lets high-rate batched inserts commit on local NVMe without waiting on S3 PUT latency, and compacts hundreds of small parts into ~1 GiB wide parts at multi-GB/s disk bandwidth before aging them once (1x) to the shared CAS pool. This eliminates EBS %iowait bottlenecks and prevents million-request S3 Tier-1 PUT storms.
When replicas share one 1x cold S3 CAS pool, does each replica still pull the same hot parts into its own NVMe cache?
No — hot parts and the NVMe cache are two separate layers in tiered_cas, and replicas never eagerly pull the shared S3 pool into NVMe. Hot parts (< cold TTL) sit on each node's local default disk and replicate over intra-AZ :9010 without touching cas_cache or S3. Aged cold parts (> cold TTL) live once (1x) in S3 CAS and attach on peers via a zero-byte CA_RELINK_COOKIE pointer with 0 bytes downloaded. Each *gd Spot reader's local NVMe SSD (measured at 1.8 GB/s direct-IO vs 131–149 MB/s on baseline gp3, 3.4x larger at $0 EBS cost) acts strictly as a lazy SLRU byte-range cache for cold S3 columns that NLB queries actually scan.
Stay in the loop — production notes on AI agents, workflows, and SRE.
Low volume — new posts and curated reading lists. Unsubscribe anytime.