Three Generations of a 30 TB/Day ClickHouse Fleet: Non-CAS vs. CAS, What Broke, and a Blueprint for Self-Managed S3
ClickHouse on AWS: Self-Managed from First Principles · Part 44
Most architectural guides for self-managed ClickHouse stop at the same
sentence: “Attach an Amazon Simple Storage Service (S3) storage policy so
cold data spills to object storage, add a local cache, and you have separated
compute from storage.”
On a whiteboard, that sounds finished. At 30 TB/day of raw log ingestion
(~1.6 TB/hour, compressing 15.2× with Zstandard (ZSTD) down to
~1.97 TB/day on disk), that whiteboard design hits three walls in quick
succession: first an S3 request-rate and disk-throughput wall, then a compute
and memory wall on background merges, and finally the replica copy
multiplier — where scaling out to 9 or 10 replicas quietly stores 9 or 10
complete copies of your cold S3 warehouse.
Over the past week, while load-testing the cluster behind this series at 30 TB/day and 100 TB/day, we evolved the exact same workload through three distinct fleet generations:
- Generation 1 — Naive Stock S3 (
26.8LTS): Premature cold-tier spills (~35wide S3 objects per part ×Rreplicas) and a hidden125 MB/sElastic Block Store (EBS) cap stalled merges within hours. - Generation 2 — Tuned Non-CAS Upstream (
26.8.36.3LTS +chinitv0.1.1): Hot-disk,S3Queue, Keeper, and merge knobs tuned from ClickHouse C++ source — proven across a 1-hour soak (792.4Mrows /1.6 TB,0failed batches,0–1slag), yet structurally capped byR×cold S3 copies and Single-Availability-Zone (Single-AZ) lock-in. - Generation 3 — Shared-Pool CAS (
26.6.8.20004.altinityantalya+chinit v0.2.7+): Altinity Antalya’s experimental Content-Addressed Storage (CAS) cold pool — collapsing cold S3 storage to1.00×and cold reader backfills to2.3–3.5 seconds, while exposing seven real operational trade-offs we had to diagnose directly in the C++ source.
This post compares the non-CAS and CAS fleets side by side without vendor spin: what CAS genuinely solves, what CAS does not touch, the seven sharp edges you sign up for in production, and a single-flag, 5-layer blueprint (plus an interactive break-even calculator) to decide which generation fits your cluster.
Key takeaways
- Fix hot-tier physics before touching CAS. Lifting writer hot disks from AWS’s
125 MB/sgp3default to500 MB/s(+$20/mo), keeping parts compact below1 GiB, and sizing writer CPUs at~1 vCPU per 8.8 MB/sof raw ingest ×2×merge headroom eliminated80%of our stability issues before CAS entered the picture. - CAS eliminates two structural taxes non-CAS cannot: it collapses cold S3 storage from
R×copies to1.00×(verified live: one3.67 GiBcold part across9replicas occupied3.68 GiBtotal in S3 instead of33.03 GiB, relinking onto new Spot readers in2.29–3.47swith0blob bytes copied) and zeroes out$0.02/GBcross-AZ transfer fees on cold reads via a regional S3 Gateway Endpoint. - CAS is an engineering trade-off, not a free lunch: it requires an experimental non-LTS build (
Altinity Antalya 26.6), a brand-new never-versioned S3 bucket with shared blast radius, exact 36-byte deterministic server UUIDs on ASG replacements,2×local cache wrappers during migration, and careful hot/cold merge separation. - CAS saves zero hot-tier CPU, RAM, or hot
:9010bandwidth. LiveS3Queueparsing, synchronous Materialized Views (MVs), localgp3merges, and hot-part replication all execute before parts ever reach CAS.
How the Fleet Evolved: Three Generations Under Load
Follow the cold-part path across all three generations: TCP :9010 peer transfers vs. $0/GB S3 Gateway Endpoint writes and zero-copy hash relinks.
You cannot evaluate CAS fairly against a misconfigured baseline. Each generation
below ran the same synthetic CloudTrail/ELB NDJSON workload (200 MB staged
files via S3Queue + 3 synchronous summary Materialized Views).
Generation 1: Naive Stock S3 (What Breaks First)
Our initial port deployed stock ClickHouse 26.8 with a 64 GiB gp3 hot disk
spilling to per-replica S3 prefixes (data/tiered/{replica}/) across a 10-slot
ASG ceiling (5 writers + 4 readers + 1 backup). Because upstream
deprecated allow_remote_fs_zero_copy_replication after garbage-collection
races (covered earlier in this series),
every replica owned a private S3 tree.
Under 30–60 TB/day load, five bottlenecks collided within two hours:
- Fleet S3
PUTamplification (~10–15×per cold byte): The64 GiBhot disk hitmove_factor = 0.1early, spilling unmerged150–500 MBparts to S3 in wide format (min_bytes_for_wide_part = 10 MiBdefault →~35S3 objects/part) where background merges rewrote them 3–5 times on every replica (~$350–$1,200+/moin ClickHouse-fleet Tier-1 S3 requests alone). - The
125 MB/sgp3iowait wall: 8-corer8g.2xlargewriters showed~28%user CPU in CloudWatch while hittingCode: 252 TOO_MANY_PARTS;iostatexposed68.79%%iowaitpinned against AWS’s default125 MB/sgp3cap. - Single-Keeper outage cascade: Replacing the lone dev Keeper dropped Raft
quorum to zero and flipped every table
is_readonly = 1. - Per-UUID
S3Queuefan-out: Omitting a shared literalkeeper_pathcreated760,000+Keeper znodes andN-1lost claim races per file. 10×cold storage:1.2 TBof cold data consumed11.99 TBin S3 across 10 replica slots.
Generation 2: Tuned Non-CAS Upstream LTS (26.8.36.3 + chinit v0.1.1)
Before adopting an experimental storage layer, we pushed 100% stock upstream
ClickHouse 26.8 LTS (cas_config.enabled = false) to its physical ceiling:
256 GiBhotgp3@500 MB/s,4,000 IOPS(+$20/mo/writer): Writer%iowaitdropped68.79% → 0.38%in 20 seconds, user CPU rose to89.4%, and local merge throughput reached494.5 MB/s.- Consolidate locally once in compact format: Setting
min_bytes_for_wide_part = 1073741824(1 GiB),max_parts_to_merge_at_once = 100, andttl_only_drop_parts = 1kept parts compact (~11objects vs.~35), merged them locally ongp3before Time-To-Live (TTL) cold moves, and dropped expired parts via free whole-partDELETEs — cutting fleet cold-tier S3 API cost to~$15–$45/mo. - Writer-only
S3Queue+ 3-Keeper Raft: ScopingS3Queueto writers with a sharedkeeper_pathandenable_hash_ring_filtering = 1balanced claims50.9% / 49.1%at<95kznodes (3–4 msKeeper latency across 3t4g.largenodes). - Empirical CPU & RAM sizing: After
m8g.large(8 GiB) OOM’d in 60s (Code: 241),m8g.2xlarge(8 vCPU / 32 GiB) completed a 1-hour30 TB/daysoak (8,240files /792.4Mrows /1.6 TBraw at220,022 rec/s,0failed batches,0–1slag) through a live ASG rolling refresh. Measuring core saturation gave our sizing rule:1writer vCPU per8.8 MB/s(0.7 TB/day) of raw NDJSON + 3 MVs ×2×merge headroom.
Where Generation 2 hits a hard structural wall:
Stock ReplicatedMergeTree still writes R× physical cold S3 copies
($4,308/mo at 3× up to $14,359/mo at 10× for 33-day retention) and pins
compute to a Single-AZ to avoid $4,800–$8,000/mo in $0.02/GB TCP :9010
cross-AZ part-fetch charges.
Generation 3: The Shared-Pool CAS Fleet (26.6.8.20004.altinityantalya + chinit v0.2.7+)
Generation 3 flips one Terraform variable: cas_config = { enabled = true }.
Terraform and chinit coordinate five automatic changes:
- Dual-Flavor AMI Selection: Switches writers, readers, and
backup-0fromFlavor=upstream(26.8.36.3) toFlavor=antalya(26.6.8.20004.altinityantalya), leaving the 3 Keepers untouched. - Dedicated Never-Versioned CAS Bucket: Creates
<cluster>-clickhouse-data-caswith versioning omitted (count = 0), a 7-dayAbortIncompleteMultipartUploadrule, and a sharedcas/tiered/prefix usingXXH3-128content hashing (no {replica}segment). - Three-Volume
tiered_casBridge Policy: Pairshot(256 GiBgp3),cold(cas_cache→cas), andcold_legacy(s3_cachefor zero-downtime in-place migration), while doubling the cache disk floor (32 → 64 GiB). - Deterministic 36-Byte Server Identity:
chinit v0.2.7+derives an exact 36-byte/var/lib/clickhouse/uuidandCH_CAS_SERVER_ROOT_IDfrom(cluster_name, claimed_ordinal)so fresh-EBS ASG replacements reclaim their CAS slot cleanly. - Source-Tuned Merge & Lease Settings: Separates hot vs. cold merge
strategy across asymmetric writers/readers and relaxes CAS GC/lease timers
(
300sGC,60s/20slease,cas_unsafe_remount_no_delay = true).
Live proof across 9 replicas: Moving a 3.67 GiB (25.4M-row) part to
VOLUME 'cold' confirmed physical 1.00× deduplication in S3:
-- system.parts (visible on writer-1 and relinked across all 9 active replicas):
table | disk_name | parts | rows | bytes
--------+-----------+-------+----------+----------
events | cas_cache | 1 | 25419480 | 3.67 GiB
-- aws s3 ls s3://<cluster>-clickhouse-data-cas/cas/ --recursive --summarize:
Total Objects: 484
Total Size: 3,950,045,875 bytes (3.68 GiB)
writer-0 published 35 blobs once (blob_put = published); peers executing
independent TTL moves skipped all 35 body uploads via mandatory HEAD checks
(CASBlobBodyPutAvoided = 35), and fresh Spot readers adopted the 3.67 GiB
part via CA_RELINK_COOKIE in 2.29–3.47 seconds with 0 blob bytes
copied.
Live Benchmark Telemetry: Non-CAS vs. Untuned CAS vs. Source-Tuned CAS
Measured 4-panel telemetry from the 1-hour 30.7 TB/day soak (640,470,621 rows across 7 nodes, $0.2467 total S3 API cost) and the 10-minute 83.85 TB/day burst (287,257,259 rows in 10m 03s drained with 0s lag).
Here is how Generation 2, our initial untuned CAS deployment (Gen 3a), and our
source-tuned CAS deployment (Gen 3b) performed under sustained 30–38 TB/day
S3Queue + 3-MV load:
| Measured Telemetry | Gen 2: Tuned Non-CAS (26.8.36.3 LTS) |
Gen 3a: Initial CAS (Untuned Defaults) | Gen 3b: Source-Tuned CAS (Antalya + chinit v0.2.7+, 1h Soak) |
|---|---|---|---|
| Sustained Ingest Pace | 220,022 rows/s (38.4 TB/day, 1.6 TB / 1h) |
212,177 rows/s (37.3 TB/day, 777 GB / 30m) |
177,821 rows/s (30.7 TB/day, 1,424.5 GiB uncomp in 1h) |
| Convergence & Staging Lag | 100% (792.4M rows), 0–1s lag, 0 failures |
100% (382.5M rows, 7 nodes), 0 backlog |
100% (640,470,621 rows across 7 nodes), 0s writer lag |
| Cold S3 Physical Copies | R× (9× = 33.03 GiB for one 3.67 GiB part) |
1.00× (3.67 GiB total across 7–9 nodes) |
1.00× (3.68 GiB total across 7–9 nodes; 15.47× ZSTD) |
| Cold Scale-Out Backfill | Full S3 download + re-upload to new {replica}/ |
2.29s – 3.47s metadata relink (0 bytes) |
2.29s – 3.47s metadata relink (0 bytes) |
| Full-Hour Fleet S3 API Cost | ~$0.25–$0.30/hr (post hot-disk tuning) |
~$0.45/hr (60s GC + 10s lease renewals) |
$0.2467 total for 1h (35,265 Tier-1 + 176,023 Tier-2 = ~$161/mo) |
| Writer CPU & RAM Usage | 65–90% on 8-vCPU m8g.2xlarge |
35–55% CPU on 16-vCPU m8g.4xlarge |
22–53% CPU, 8.44 / 61.64 GiB RAM (13.7%) → m8g.2xlarge floor |
| Reader Merge Behavior | Local hot merges (20–45s) |
63–110s avg (1,098s max) — 57% hashed to readers |
prefer_fetch_merged_part = 180s / 1 MiB + readers at ≥16 GiB RAM (m8g.xlarge class, concurrencyRatio = 2 at ≤12 GiB) |
systemctl restart Time |
~5–7s |
40–89s (waits out prior mount lease TTL) |
7s (cas_unsafe_remount_no_delay = true on own UUID) |
| Fresh-EBS ASG Roll | Clean ({replica} macro) |
Crash-looped (Code: 246 / Code: 27) |
Clean (36-byte deterministic UUID + CH_CAS_SERVER_ROOT_ID) |
Takeaway: CAS delivered 1.00× cold deduplication and ~3s scale-out
relinks out of the box; getting merges, restarts, and stateless ASG replacements
production-clean required reading the C++ source.
Side-by-Side Comparison: Non-CAS (Gen 2) vs. CAS (Gen 3)
(Scope note: every dollar figure accounts strictly for the ClickHouse fleet
itself — excluding upstream log-producer or load-generator compute and
producer staging PUTs.)
| Dimension | Gen 1: Naive Stock S3 | Gen 2: Tuned Non-CAS (26.8 LTS) |
Gen 3: Source-Tuned CAS (Antalya 26.6) |
|---|---|---|---|
| ClickHouse Binary | Upstream 26.8 LTS (GA) |
Upstream 26.8.36.3 LTS (GA) |
Altinity Antalya 26.6.8.20004 (Experimental, Apache-2.0) |
Cold S3 Copies (R replicas) |
R× (10× across 10 slots) |
R× (3×–10× per replica ceiling) |
1.00× shared content-addressed pool |
| 33-Day Cold S3 @ 30 TB/day | $14,359/mo (10 slots) |
$4,308/mo (3×) to $14,359/mo (10×) |
$1,436/mo flat (at 3, 10, or 20 replicas) |
| 7-Day Cold S3 @ 30 TB/day | $3,178/mo (10 slots) |
$953/mo (3×) to $3,178/mo (10×) |
$318/mo flat |
| Fleet Cold-Tier S3 API Cost | ~$350–$1,200+/mo (wide spills × R) |
~$15–$45/mo (compact local merge × R) |
~$15–$35/mo flat (1× PUTs + 300s GC) |
| Elastic Reader Scale-Out | Copies entire cold S3 table per reader | Copies entire cold S3 table per reader | $0 extra cold S3 bytes; 2.3–3.5s relink |
Multi-AZ Transfer ($0.02/GB) |
$4,800–$8,000/mo on hot + cold :9010 |
Pinned to Single-AZ to avoid :9010 fee |
$0/GB on cold S3 reads; hot inserts still use :9010 |
| Hot-Tier EBS & Writer CPU | 64G @ 125 MB/s (stalled) |
256G @ 500 MB/s; 48–64 writer vCPUs |
Identical (256G @ 500 MB/s; 48–64 writer vCPUs) |
| Local Cache Disk Split | 2 wrappers | 2 wrappers (32 GiB = 14 GiB each) |
4 wrappers during migration (64 GiB = 14 GiB each) |
| S3 Versioning & Blast Radius | Supported; isolated per {replica}/ |
Supported; isolated per {replica}/ |
Forbidden (Code: 48); shared pool (nightly DR mandatory) |
| Garbage Collection & ASG ID | S3 Lifecycle + {replica} macro |
ttl_only_drop_parts=1 + {replica} |
18-phase S3 GC + exact 36-byte deterministic UUID |
The Unbiased Pros and Cons of CAS
The Pros: Where CAS Changes Cluster Economics
- Replica count stops multiplying your cold storage bill (
R× → 1×). At30 TB/day(1.974 TB/daycompressed) and 33-day retention, one cold S3 copy costs$1,436/month. Non-CAS scales linearly from$4,308/mo(3×) to$14,359/mo(10×); CAS locks cold S3 at$1,436/moflat (-$34kto-$155k/year) regardless of how many Spot readers you add. - Sub-4-second cold scale-outs (
DataPartsExchange.cpp:408). When peers share a CAS pool UUID, the sender returnsCA_RELINK_COOKIE+part_manifest_v2instead of streaming column files, and the receiver stages a manifest reference (CASBlobAdoptTrusted) in2.29–3.47s. Even when replicas independently TTL-move the same hot part,CasPartWriteTxn.cppruns aHEADcheck and skips duplicate uploads (CASBlobBodyPutAvoided). $0.00/GBcross-AZ cold reads via S3 Gateway Endpoint. Unlike EC2-to-EC2:9010transfers ($0.02/GBacross AZs), regional S3 Gateway Endpoint traffic is$0.00/GBfrom every AZ in the region — unlocking cross-AZ cold query replicas and fast cross-AZ recovery.- Keeper stays completely off the blob GC path.
Unlike deprecated zero-copy replication (which bloated Keeper with per-blob
znodes and locks), CAS stores manifests, reference logs, and mount leases in
S3 (
cas/tiered/gc/), keeping our 3 Keepers at3–4 mslatency.
The Cons: Seven Sharp Edges Diagnosed from Source
- Experimental non-upstream binary (
Altinity Antalya 26.6) with no format stability promise.metadata_type = casis not in upstream ClickHouse26.8LTS. You cannot mix26.8and26.6server settings (s3_allow_server_credentials_in_user_queriescrashes26.6withCode: 115 UNKNOWN_SETTING), and Altinity explicitly notes on-S3 CAS metadata has no cross-version compatibility guarantee yet. - Stateless ASG replacements
Code: 246crash-loop — down to a trailing\nbyte. CAS binds each replica’s mount slot (gc/server-roots/<srid>/ownerinCasServerRoot.cpp:712) to/var/lib/clickhouse/uuid. Three failure modes await ASG fleets:- Distro auto-start race: Ubuntu’s
.debpackage startsclickhouse-serverduring image bake, minting a random UUID unless masked (systemctl mask clickhouse-serverin Packer). - Fresh-EBS replacement: A replacement
writer-0with a new random UUID crashes on startup withCode: 246 CORRUPTED_DATA("CAS server-root is owned by a different server") unless bothCH_CAS_SERVER_ROOT_IDand/var/lib/clickhouse/uuidare derived deterministically from(cluster_name, ordinal). - The 37th-byte newline trap (
ServerUUID.cpp:44–45): ClickHouse parses/var/lib/clickhouse/uuidviareadUUIDText(uuid, in); assertEOF(in);. Writinguuid + "\n"(37bytes) throwsCode: 27 CANNOT_PARSE_INPUT_ASSERTION_FAILEDand silently overwrites the file with a randomUUIDv4— priming a delayedCode: 246crash on the next ASG roll untilchinit v0.2.3wrote exact 36-byte files (the newline-bug fix remains load-bearing today —chinit v0.2.7keepsServerUUID::loadtolerant of trailing newlines).
- Distro auto-start race: Ubuntu’s
- S3
"Suspended"versioning still fails (Code: 48), and blast radius is shared. If a bucket ever had S3 Versioning enabled — even after settingStatus="Suspended"— S3 permanently createsnulldelete markers onDELETE, causingCasProbeto refuse startup withCode: 48 NOT_IMPLEMENTED. You need a brand-new never-versioned bucket, and because all replicas share its blobs, pool corruption or accidental prefix deletion hits every replica at once. - Zero hot-tier compute savings — and hot
:9010replication still pays cross-AZ fees.S3Queueparsing, synchronous MVs, and localgp3merges consume identical CPU and RAM, and newly inserted hot parts (~1.97 TB/daycompressed) still replicate over:9010(~$1,180/moper cross-AZ hot replica at$0.02/GBunless writers stay in one AZ per shard). - The
execute_merges_on_single_replica_time_thresholdasymmetrical-fleet trap. InReplicatedMergeTreeMergeStrategyPicker.cpp:57–104, settingexecute_merges_on_single_replica_time_threshold = 3600on a two-volume policy hashes hot local merges across all replicas — assigning57%(4/7) of heavy ZSTD merges to2-vCPUreaders (63–110savg,18.3mmax) while16-vCPUwriters logged220Code: 234 NO_REPLICA_HAS_PARTerrors. Keepexecute_merges_on_single_replica_time_threshold = 0on hot tiers, setprefer_fetch_merged_part_time_threshold = 180/size_threshold = 1 MiBso readers merge tiny parts locally and fetch large merges from writers, and gateremote_fs_execute_merges_on_single_replica_time_thresholdstrictly onisContentAddressed()cold disks. - Live policy migration (
checkCompatibleWith) doubles cache wrappers. Switching live tables fromtieredtotiered_casrequires keepings3_cacheascold_legacy, doubling active cache wrappers from2 → 4and requiring at least a64 GiBcache disk (14 GiBper wrapper) or local NVMe. - S3 mount-lease and 18-phase GC state machine overhead.
Default CAS timers (
60sGC,30s/10slease,cas_unsafe_remount_no_delay = false) burn idle Tier-1PUTs and stallsystemctl restartfor40–89s. Tuning them cut restarts to7sand idle GC ops by5×, while addingsystem.cas_mounts(wedged_namespace_count,gc_fenced) to SRE monitoring.
A Blueprint for Future Self-Managed ClickHouse Installations
For Bring-Your-Own-Cloud (BYOC) SaaS vendors, strict data-sovereignty
deployments, or 10–100+ TB/day observability fleets optimizing Cost of Goods
Sold (COGS), our three-generation evolution distills into a 5-layer
architecture where non-CAS vs. CAS is one declarative toggle:
1. Design CAS as a Single-Flag Toggle (cas_config.enabled)
Generation 2 and Generation 3 share 98% of our Terraform and chinit code:
# Flipping enabled = true/false automatically adapts:
# 1. Golden AMI flavor (Flavor=antalya 26.6 vs Flavor=upstream 26.8 LTS)
# 2. Never-versioned <cluster>-clickhouse-data-cas S3 bucket + IAM scopes
# 3. Cache volume floor (>= 14 GiB per active cache wrapper)
# 4. Storage policy ('tiered_cas' with cold_legacy vs 'tiered')
# 5. chinit 36-byte deterministic UUID + live storage_policy convergence
cas_config = {
enabled = true
}
Launch small clusters on Generation 2 (26.8 LTS non-CAS) and flip
cas_config.enabled = true once retention and replica count justify shared S3.
2. Source-Verified CAS & MergeTree Reference Configuration
Every snippet below is pure XML and SQL ready to copy into test environments:
<!-- /etc/clickhouse-server/config.d/storage.xml (CAS disk excerpt) -->
<cas>
<type>object_storage</type>
<object_storage_type>s3</object_storage_type>
<metadata_type>cas</metadata_type>
<cas_server_root_id from_env="CH_CAS_SERVER_ROOT_ID" />
<endpoint from_env="CH_S3_ENDPOINT" />
<endpoint_subpath>cas/tiered/</endpoint_subpath>
<use_environment_credentials>true</use_environment_credentials>
<skip_bucket_creation>true</skip_bucket_creation>
<cas_gc_interval_sec>300</cas_gc_interval_sec>
<cas_mount_lease_ttl_ms>60000</cas_mount_lease_ttl_ms>
<cas_mount_renew_period_ms>20000</cas_mount_renew_period_ms>
<cas_unsafe_remount_no_delay>true</cas_unsafe_remount_no_delay>
<cas_blob_hash>xxh3-128</cas_blob_hash>
<cas_blob_hash_allow_new>true</cas_blob_hash_allow_new>
</cas>
Pair that disk with these MergeTree settings on two-volume (hot local gp3
→ cold cas_cache) tables:
SETTINGS
storage_policy = 'tiered_cas',
min_bytes_for_wide_part = 1073741824,
max_parts_to_merge_at_once = 100,
ttl_only_drop_parts = 1,
execute_merges_on_single_replica_time_threshold = 0,
prefer_fetch_merged_part_time_threshold = 180,
prefer_fetch_merged_part_size_threshold = 1048576;
3. The Five Non-Negotiable Operational Rules
- Deterministic 36-byte cattle identity: Lease bounded ordinals
(
writer-0..N,reader-0..M), maskclickhouse-serverin Packer, and seed an exact 36-byte deterministic UUID before first start (see Con #2). - Merge locally on
500 MB/sgp3before cold TTL: Consolidate inserts into≥1 GiBcompact parts on256–384 GiBgp3before async1–2dTTL moves write them to CAS once (perform_ttl_move_on_insert = false). - Disposable NVMe cache + enforced S3 Gateway Endpoint: Auto-stripe local
*gdNVMe SSDs (or≥64 GiBgp3) forcas_cache, and verify at plan time that every subnet route table attaches to the regional S3 Gateway Endpoint. - Strict writer/reader isolation +
/replicas_statusNLB gate: DropS3Queueimmediately afterSyncSchemaon non-writers, run non-burstable4-vCPUnon-burstable Graviton4 readers sized at ≥ 16 GiB RAM (e.g.m8g.xlarge;c8g.xlarge’s 8 GiB was the source of the readerCode: 241), withconcurrencyRatio = 2under12 GiB, and checkHTTP :8123/replicas_status(delay ≤ 300s,deregistration_delay = 30s) on NLB target groups. - Nightly blast-radius isolation: Wake an off-quorum
backup-0replica nightly (~$1.50/mocompute) to archiveclickhouse-backupsnapshots into a separate, versioned S3 bucket before stopping itself.
Production-Readiness Scorecard (7.4/10 → 9.4/10)
Our 1-hour 30.7 TB/day (640.47M-row) soak with 5-minute EC2-to-S3 snapshots
surfaced four subtle bottlenecks that short smoke tests miss — all closed in
chinit and our ClickHouse C++ branch:
| Dimension | Initial Score | Remediated | What the 1-Hour 30.7 TB/day Soak Exposed & Fixed |
|---|---|---|---|
| 1. Ingest & Storage Efficacy | 9.0 / 10 |
9.5 / 10 |
6,661 files / 640.47M rows (1,424.5 GiB → 92.1 GiB = 15.47× ZSTD) at 0s lag and $0.2467 total S3 API cost/hr (~$161/mo); shortened cold TTL to 1–2d. |
| 2. CPU Sizing & Balance | 6.5 / 10 |
9.4 / 10 |
m8g.4xlarge writers ran at 22–53% CPU while 2-vCPU t4g.large readers hit 100%; baseline writers right-sized to m8g.2xlarge with a vertical-autoscale ladder, readers upgraded off burstable credits to 4-vCPU Graviton4. |
| 3. Memory Stability | 7.0 / 10 |
9.4 / 10 |
Writers peaked at 13.7% RAM (8.44 GiB); new 8 GiB readers hit Code: 241 because background_merges_mutations_concurrency_ratio = 4 allowed 32 concurrent merges against a ~6 GiB server cap. Fix in chinit v0.2.7: HardwareSizingOverlayXML now caps concurrencyRatio = 2 whenever memBytes <= 12 GiB, and compute-reader floors were raised to ≥16 GiB. |
| 4. Network & Replication | 7.5 / 10 |
9.3 / 10 |
200–444 MB/s burst over :9010 + free S3 Gateway Endpoint. Live logs showed stale Keeper DDL-queue znodes (%2D/%2E-encoded) for decommissioned ASGs; dead-replica-gc now URL-decodes them (chinit v0.2.7), and health_gate re-converges /etc/hosts + DNS + remote_servers between retries. |
| 5. Load Balancing & Routing | 7.0 / 10 |
9.4 / 10 |
Switched NLB target groups from TCP :8123 to HTTP :8123/replicas_status (300s / 120s lag thresholds) and cut deregistration_delay 300s → 30s. |
| Overall Fleet Readiness | 7.4 / 10 |
9.4 / 10 |
Production-ready for sustained 30+ TB/day ingestion. |
What the 10-Minute 83.85 TB/day Burst Exposed After the Soak
The follow-up burst (2,987 files, 287,257,259 rows, 476,598 rec/s, 10m 03s) re-confirmed the fixes against a harder target — and caught one more:
Code: 241on8 GiBreaders is a concurrency bug, not a capacity bug. The sampler showed28concurrent merges +24fetches onreader-0;background_pool_size × background_merges_mutations_concurrency_ratiowas8 × 4 = 32on a box whosemax_server_memory_usage_to_ram_ratio = 0.9caps it at~6.0 GiB.chinit v0.2.7caps the ratio at2below12 GiBphysical RAM.%2D/%2EKeeper znodes accumulate silently in/clickhouse/<cluster>/task_queue/replicas.DDLWorkerandDNSResolverlogged warnings for hosts retired weeks earlier because the reaper compared raw entries without URL-decoding ClickHouse’s percent-encoded path segments.dead-replica-gcinchinit v0.2.7decodes them; 21 stale znodes were removed by hand onwriter-0to confirm the class was gone.opentelemetry_span_logcrashed fresh nodes (Code: 36) becausesystem.opentelemetry_span_logships with an explicit<engine>block in the stockconfig.xml; settingopentelemetry_span_log/partition_by+ttlalongside it is illegal. The engine belongs inline in theengine=string — fixed in bothchi.yaml.tftplflavors inchinit v0.2.6/v0.2.7and in the live template.- Writers hit
92–99%CPU and13–21 GiBRAM during the burst — comfortable on16 vCPU / 64 GiBm8g.4xlarge, but the reasonwriter_min_typestays atm8g.4xlargewhile steady-state dropped tom8g.2xlargeafter the soak. - NLB
:8123/replicas_status+min_absolute_delay_to_close=300/min_relative_delay_to_close=120kept lagging autoscaled readers out of rotation without a single client-visible error;deregistration_delaydropped300s → 30s. - A clean
2fc6bb4build (chinit v0.2.7) is the tag that carries all of the above (v0.2.7cut after the burst, before production promotion).
Follow-Up Chaos Run (chinit v0.2.8 + Antalya 26.8): 100% Spot Writers, NVMe Readers & The Hot/Cold Cache Boundary
To stress-test how far cost and elasticity can be pushed, we next moved both ASGs onto 100% EC2 Spot — writers across m8g / m7g / m6g.4xlarge (writer_min = 1, quorum = 2) and readers onto NVMe-equipped *gd Graviton pools (m8gd / m7gd.xlarge + c8gd.2xlarge) with a role-scoped reader_cache_on_nvme_instance_store = true flag (so readers drop /dev/sdc gp3 cache and mount their 217 GiB local NVMe SSD as 4 × 48.65 GiB wrappers, while EBS-only writers keep their 64 GiB gp3 cache) — and ran a second 81.86 TB/day (466,820 rec/s, 281.30M-row) 10-minute burst through live AWS Spot interruptions:
- Exact 6-node parity (
334,739,767rows,0lost) across6real AWS Spot reclaims and-46%cost: Across a 100-minute window (15Spot launches,1writer reclaim,5single-AZm8gd/m7gdreader reclaims, plus a mid-burst3 → 2 → 4writer swing where 2 Spot writers solo-carried81.86 TB/dayat99.6%CPU and1slag), staging drained to0in49s,chinit-spot-watchdrained every 2-minute ITN in<0.6s,/replicas_statusheld backfilling readers out of NLB rotation untillag = 0s, and live cluster run-rate dropped 46% ($3.26/hr → $1.75/hr). - Why Single-AZ
*gdSpot needs Multi-AZ + CAS: Pinning*gdSpot readers to one AZ hit 5 reclaims and a3.5mUnfulfillableCapacitypinch. Because CAS cold parts relink across AZs via S3 at$0.00/GB, enabling Multi-AZ + widening*gdfamilies (r8gd/r7gd/c7gd) expands Spot pool diversity6×. - Answering “Does each replica still pull the same hot parts into its own NVMe cache?”: Direct inspection of
system.filesystem_cache(FilesystemCacheBytes = 0on fresh194.6 GiBcapacity) anddddirect-IO benchmarks (1.8 GB/sNVMe read vs.131–149 MB/sbaselinegp3cache =~13×faster) confirmed an essential architectural distinction:- Hot parts (
< cold TTL) never touch NVMecas_cache— they live on each replica’s<hot>defaultdisk and sync over:9010. - Cold parts (
> cold TTL) live1×in S3 CAS and relink with0bytes copied — replicas never eagerly download the shared S3 pool onto NVMe. - NVMe
cas_cacheis strictly a lazySLRUcold-read cache populated only for cold column byte-ranges thatSELECTqueries actually scan. If your cold-move TTL is7 DAY, NVMecas_cachesits0 Bidle for the first week — lower cold TTL to1–2 DAYTO VOLUME 'cold'to cut per-replica hotgp3size by~75–85%and put days2–33onto1×S3 CAS +1.8 GB/sNVMe immediately.
- Hot parts (
Break-Even Math & Interactive Calculator: Which Generation Should You Run?
Monthly cold-storage savings (ΔC_cold) when switching from Generation 2
(non-CAS, R× copies) to Generation 3 (CAS, 1× shared pool) follow one
equation:
R = data-bearing replica slot ceiling · Vraw/day = daily raw ingest (GB) · Dcold = cold retention days · c = ZSTD compression ratio (~12–15×) · PS3 = blended S3 price (~$0.022–$0.023/GB-mo)
Interactive Non-CAS vs. CAS Cold-Storage Calculator
Plug in your own cluster numbers below (or share a direct permalink to any scenario):
# Cold-Storage Cost Estimator
| Your Situation | Recommended Choice | Why |
|---|---|---|
Ingest < 2–5 TB/day, 2–3 replicas in one AZ, strict LTS or S3-versioning mandate |
Gen 2: Tuned Non-CAS (26.8 LTS + chinit) |
Cold S3 is only $150–$600/mo; 500 MB/s gp3 + 1 GiB compact parts deliver GA stability, S3 versioning, and isolated per-replica blast radius. |
Ingest 10–100+ TB/day, BYOC VPCs, elastic 4–10+ Spot readers, or cross-AZ cold serving |
Gen 3: Source-Tuned CAS (Antalya + chinit v0.2.7+) |
Saving $3,000–$15,000+/mo in duplicate S3 storage + 2.3–3.5s zero-copy reader scale-outs justifies pinning Altinity Antalya and running nightly isolated backups. |
| No appetite to own Keeper quorums, AMI pipelines, or storage GC runbooks | ClickHouse Cloud (SharedMergeTree) / Altinity.Cloud |
Shared object storage, stateless compute scaling, and vendor Service-Level Agreements (SLAs) out of the box without operating the control or storage planes yourself. |
Self-managed ClickHouse on object storage is crossing an important inflection
point. By fixing hot-tier merge physics first, treating CAS’s trade-offs
honestly, and automating deterministic VM identity in
chinit, 1× shared S3 storage with
elastic, zero-copy read scaling is now a blueprint any infrastructure team can
evaluate with clear math — and toggle with a single flag when scale demands it.
FAQ
What is the real difference between a non-CAS ClickHouse S3 cluster and a CAS cluster?
Stock upstream ReplicatedMergeTree writes an isolated copy of every cold S3 part per replica ({replica}/), multiplying cold storage bytes and PUT requests by R. Altinity Antalya's Content-Addressed Storage (CAS) stores cold blobs once by their XXH3-128 hash in a shared S3 pool so replicas relink cold parts in ~2-3 seconds with 1.00x total S3 storage.
Does CAS reduce ClickHouse writer CPU, memory, local EBS disk, or hot-tier cross-AZ traffic?
No — CAS only deduplicates cold S3 data after parts merge and age off the hot disk. Live inserts, S3Queue parsing, synchronous Materialized Views, local gp3 merges, and hot peer-to-peer TCP :9010 replication still consume identical CPU, RAM, and hot network bandwidth.
What are the biggest operational risks of enabling CAS on an Auto Scaling Group fleet?
Every replica shares one never-versioned S3 pool (requiring independent nightly backups and a brand-new bucket that never had S3 Versioning enabled or suspended), and stateless ASG replacements crash with Code 246 unless your bootstrapper seeds an exact 36-byte deterministic server UUID. You also depend on Altinity's experimental Antalya build rather than stock upstream ClickHouse LTS.
When should a team stay on stock non-CAS upstream ClickHouse instead of CAS?
Stay on tuned non-CAS upstream ClickHouse 26.8 LTS when daily ingest is under ~2-5 TB/day, your replica ceiling stays at 2-3 nodes in a single AZ, or compliance mandates GA upstream binaries and S3 Bucket Versioning. Below ~$300-$600/month in cold S3 savings, operating a shared experimental pool is rarely worth the added operational surface.
What does the blueprint for future self-managed ClickHouse installations look like?
Combine ephemeral ASG compute given deterministic identities via chinit, a 500 MB/s gp3 hot disk for local >=1 GiB merges, disposable local NVMe SSDs for S3 cache, and a 1x shared CAS pool behind a free regional S3 Gateway Endpoint — paired with an on-demand backup replica writing to an isolated versioned bucket. Making CAS a single declarative toggle lets teams start on GA non-CAS LTS and enable CAS only when scale justifies it.
Stay in the loop — production notes on AI agents, workflows, and SRE.
Low volume — new posts and curated reading lists. Unsubscribe anytime.