The Math of ClickHouse CAS on AWS: How Content-Addressed S3 Cuts Cold Storage from R× to 1×
ClickHouse on AWS: Self-Managed from First Principles · Part 42
When teams first wire self-managed ClickHouse to Amazon S3, there is a very common misconception: “We attached an S3 Gateway VPC Endpoint, so S3 traffic is free — why is S3 still our biggest bill?”
The answer comes down to separating the wire from the warehouse — and making sure nothing on your fleet is secretly bypassing the wire in the first place.
In Free Is a Route, Not a Setting,
we showed why an S3 Gateway Endpoint (com.amazonaws.<region>.s3) is
mandatory at 30 TB/day: it keeps VPC-to-S3 bytes off the $0.045/GB NAT
Gateway, saving ~$2,700–$9,300/month.
But even when every byte rides the free S3 Gateway Endpoint at $0.00/GB,
stock open-source ClickHouse (ReplicatedMergeTree) has a structural
multiplier: because upstream zero-copy replication
(allow_remote_fs_zero_copy_replication)
was deprecated, every replica writes its own physical copy of every
cold part to S3.
This post lays out the exact mathematics of that multiplier at 30 TB/day
(912,000 GB/month), first closes the remaining places where S3 calls
can still leak past the Gateway Endpoint or multiply S3 API charges in a
real stack, and then walks through how Altinity Antalya’s
Content-Addressed Storage (CAS, metadata_type = cas) collapses cold
storage from R× to 1× — taking our end-to-end log platform
COGS down to $0.0026/GB (0.26¢/GB) at 7-day retention and
$0.0038/GB (0.38¢/GB) at 33-day retention.
1. First: If S3 Is Still Costing You on the Wire, Fix the Silent Leaks
Before you optimize storage copies, verify that your S3 Gateway Endpoint
is actually catching 100% of your S3 traffic. An S3 Gateway Endpoint
injects a regional prefix-list route (for example, the regional S3 prefix
list for com.amazonaws.<region>.s3) into the route tables you attach it to:
Destination Target Cost
pl-xxxxxxxx (S3) vpce-xxxxxxxxxxxxxxxxx $0.00/hr, $0.00/GB (Gateway Endpoint)
0.0.0.0/0 nat-xxxxxxxxxxxxxxxxx $0.045/hr + $0.045/GB processed
If an S3 request targets any IP outside the regional S3 prefix list, or
originates from any subnet whose route table lacks that entry, it falls
through to 0.0.0.0/0 and bills $0.045/GB on the NAT Gateway with zero
errors. When we audited our bootstrapper (chinit),
our Terraform modules, and our live VPC route tables, we found three subtle
leaks:
- Region-less
aws s3 cpin stage-0user-data.sh: Inchinit’s stage-0 cloud-init wrapper (user-data.sh.tftpl), the node fetches its role spec beforechinitstarts:# BEFORE: no --region and no AWS_REGION in cloud-init's root env! aws s3 cp "s3://${bundle_bucket}/${bundle_prefix}/$CHINIT_SPEC_KEY" "$CHINIT_SPEC_PATH"When
AWS_REGIONandAWS_DEFAULT_REGIONare unset inuser-data, AWS CLI v2 defaults to the globals3.amazonaws.com(us-east-1) endpoint first. Becauseus-east-1S3 IPs are not in aus-west-2endpoint’s prefix list, the lookup exits via the NAT Gateway (and fails outright in air-gapped NAT-less VPCs). ExportingAWS_REGION="${region}"and passingaws s3 cp --region "${region}"pins stage-0 tos3.<region>.amazonaws.comon the free Gateway Endpoint. - Region fallback in
clickhouse-backup’sconfig.yml:chinit’sconfigureBackuprenderedregion:fromb.Identity.Regionalone; if identity metadata hadn’t populated the field,region:rendered blank andclickhouse-backupdefaulted tous-east-1. Falling back acrossb.Identity.Region → b.Spec.Region → b.Meta.Regionguaranteesclickhouse-backupalways hits the regional S3 endpoint. - Sibling VPCs & VPC-attached Ingest Lambdas missing the Gateway Endpoint:
Our dedicated ClickHouse VPC had the S3 Gateway Endpoint attached,
but a sibling workload VPC had no S3 Gateway Endpoint attached to its
private route table, sending every S3 byte through its NAT Gateway.
Similarly, our optional
lambda_vpc_configmodule for VPC-attached Ingest Lambdas previously documented using NAT or paid Interface Endpoints for S3 instead of attaching a freeGatewayendpoint to the Lambda’s route tables.
What the S3 Gateway Endpoint Does Not Make Free
Once those leaks are closed, network transport to S3 is $0.00/GB.
What remains on the S3 bill are the two things a Gateway Endpoint never
discounts:
- Physical S3 Storage (
TimedStorage-ByteHrs):- S3 Standard:
$0.023 / GB-month($23.00 / TB-month) - S3 Standard-IA (after 30 days):
$0.0125 / GB-month($12.50 / TB-month)
- S3 Standard:
- S3 API Requests (
Requests-Tier1&Requests-Tier2):- Tier-1 (
PUT,COPY,POST,LIST):$5.00 per 1,000,000 requests($0.005 / 1,000) - Tier-2 (
GET,HEAD,SELECT):$0.40 per 1,000,000 requests($0.0004 / 1,000) - S3
DELETE:$0.00(Free)
- Tier-1 (
And that is where ClickHouse’s cold-tier replication architecture
determines whether your S3 bill is $318/month or $14,359/month.
2. The R× Cold-Storage Problem in Stock ClickHouse
In our tiered storage layout, new
inserts land on the local hot gp3 volume (default, 256 GiB), merge
locally in the background, and move to the S3 cold volume after the hot
TTL window (or when disk pressure trips move_factor = 0.1).
In stock ClickHouse 26.8 LTS, each replica must isolate its S3 objects
under its own {replica} prefix:
<s3>
<type>s3</type>
<endpoint from_env="CH_S3_ENDPOINT" />
<endpoint_subpath>data/tiered/{replica}/</endpoint_subpath>
</s3>
Why? Because without a content-addressed shared metadata layer, two replicas pointing at the same S3 path will collide on part names and delete objects out from under each other during background merges and garbage collection.
The consequence is brutal: every data-bearing replica slot stores a complete, independent physical copy of the cold dataset in S3.
During our 30 TB/day and 60 TB/day soak tests on a 1-shard cluster
with writer_max = 5, max_readers = 4, and backup = 1 (10 replica
slots), we ran a full list-objects-v2 aggregation across
s3://<clickhouse-data-bucket>/data/tiered/:
Replica Prefix (data/tiered/<replica>/) |
Stored Cold Bytes | S3 Object Count |
|---|---|---|
writer-0 |
3.08 TiB |
1,700,000 |
writer-1 |
3.40 TiB |
2,570,000 |
writer-2 |
2.98 TiB |
2,140,000 |
writer-3 |
2.74 TiB |
2,130,000 |
writer-4 |
3.33 TiB |
2,690,000 |
reader-0 |
1.70 TiB |
815,563 |
reader-1 |
2.19 TiB |
~1,100,000 |
reader-2 |
2.39 TiB |
~1,200,000 |
reader-3 |
1.04 TiB |
~520,000 |
backup-0 |
0.38 TiB |
363,896 |
| Total Across 10 Replica Prefixes | 23.23 TiB |
~15.23M objects |
Even though S3 Standard is already 3-AZ durable (99.999999999% eleven-nines durability), stock ClickHouse stored 10 physical copies of the same compressed log parts in the same S3 bucket. Those extra 9 copies buy zero additional durability — they are pure storage waste.
3. The Exact Cost Math: Stock R× vs. Sharding vs. CAS 1×
Let’s write down the exact equations from our verified us-west-2 billing
model so you can plug in your own daily volume V, compression ratio c,
retention days D, shard count S, and total data-bearing nodes R.
Step 1: Daily Compressed Volume Entering ClickHouse
Let:
V = 30 TB/day(912,000 GB/monthover a30.4-day month) of raw uncompressed normalized logs.c = 15.2×measured ClickHouse ZSTD compression ratio (Delta(8) + ZSTD(1)on timestamps,ZSTD(3)onMap(String, String)attributes).
Then the compressed volume written per day for one complete copy of the dataset is:
Step 2: How Many Physical Copies (r) Land in S3?
In a cluster with S shards and R total data-bearing nodes
(r = R / S replicas per shard):
- Each shard holds
1 / Sof the dataset, replicated acrossr = R / Sreplicas. - Therefore, total cold S3 storage across all
Sshards is:
- Unsharded (
S = 1) without CAS:r = Rfull copies (10×whenR = 10). - Sharded (
S = 3) without CAS:- Zero-lag (
6 writers + 3 readers= 3 replicas/shard):r = 3full copies (3×). - Bounded-lag (
3 writers + 3 readers= 2 replicas/shard):r = 2full copies (2×).
- Zero-lag (
- With Altinity Antalya CAS (
metadata_type = cas): Every replica of a shard publishes references into one shared content-addressed pool (cas/tiered/orcas/<tenant>/), where each blob is stored under its content hash:
Step 3: Cold S3 Monthly Cost Equation
With S3 Standard priced at $23.00/TB-month for days 1–30
and S3 Standard-IA priced at $12.50/TB-month for days >30:
Evaluating at V = 30 TB/day (V / c = 1.974 TB/day):
- For
D = 7days raw retention (min(7, 30) = 7,max(0, −23) = 0;lifecycle_days_to_ia = 0to avoid the 30-day Standard-IA minimum duration charge):Cost per copy (7d) = 1.974 TB/day × 7 days × $23.00/TB = 13.82 TB × $23.00 = $317.76/month - For
D = 33days raw retention (30days Standard +3days Standard-IA):Cost per copy (33d) = 1.974 TB/day × (30 × $23.00 + 3 × $12.50) = 1.974 × $727.50 = $1,435.86/month
Multiply by r across the five real cluster topologies:
| Cluster Topology (@ 30 TB/day) | S3 Copies (r) |
7-Day Cold S3 $/mo | 33-Day Cold S3 $/mo | CAS Savings @ 7d | CAS Savings @ 33d |
|---|---|---|---|---|---|
Unsharded 10-Slot (6W+3R+1B, no CAS) |
10× |
$3,178/mo |
$14,359/mo |
-$2,860/mo (-90%) |
-$12,923/mo (-90%) |
Unsharded Bounded-Lag (3W+2R+1B, no CAS) |
6× |
$1,907/mo |
$8,615/mo |
-$1,589/mo (-83%) |
-$7,179/mo (-83%) |
Sharded GA Zero-Lag (3 shards × 3 replicas) |
3× |
$953/mo |
$4,308/mo |
-$635/mo (-67%) |
-$2,872/mo (-67%) |
Sharded GA Bounded-Lag (3 shards × 2 replicas) |
2× |
$636/mo |
$2,872/mo |
-$318/mo (-50%) |
-$1,436/mo (-50%) |
Altinity Antalya CAS (cas_config.enabled = true) |
1× |
$318/mo |
$1,436/mo |
Baseline (1×) |
Baseline (1×) |
Want to model your own daily TB, retention window, and replica count? Try the live Interactive Non-CAS vs. CAS Cold-Storage Calculator in Part 44.
4. Three More Cost Wins CAS Unlocks Beyond Storage Bytes
A. S3 API Request Reduction (PUT Once, Relink R − 1 Times)
Even after applying our five S3 API storm fixes
(256 GiB hot gp3, min_bytes_for_wide_part = 1 GiB cutting objects
per part from ~35 to ~11, ttl_only_drop_parts = 1, shared S3Queue
keeper_path, and enable_hash_ring_filtering = 1), non-CAS replicas
still pay:
Plus, every time an Auto Scaling Group scales out a fresh reader or
replaces a spot instance, a non-CAS replica must download all cold parts
and re-PUT them into its own {replica} prefix.
With CAS enabled alongside single-replica merge coordination
(execute_merges_on_single_replica_time_threshold = 3600):
sequenceDiagram
autonumber
participant W as writer-0
participant K as ClickHouse Keeper
participant S3 as S3 CAS Pool (cas/tiered/)
participant R as reader-1 (or new scale-out node)
W->>W: Merge hot parts locally on 256 GiB gp3
W->>S3: PUT part blob once under content hash (cas/tiered/<hash>)
W->>S3: PUT server manifest ref (cas_server_root_id = writer-0)
W->>K: Commit part entry to replication log
K-->>R: Replication log notification
R->>S3: HEAD blob by hash + publish ref (cas_server_root_id = reader-1)
Note over W,R: Zero blob bytes copied or re-uploaded by reader-1!
Our empirical S3 API formula at V TB/day (assuming ≥ 150 MB
staged files and ≥ 256 GiB hot disks):
With CAS (r = 1): S3 API $/mo ≈ $60 + V × $2.00/TB = $110–$120/mo @ 30 TB/day
B. Eliminating the $4,800–$8,000/month Multi-AZ Cross-AZ Transfer Tax
Why can ClickHouse Cloud run across 3 Availability Zones without a massive
AWS cross-AZ data-transfer bill, while stock self-managed
ReplicatedMergeTree gets crushed if you spread a shard across us-west-2a,
2b, and 2c?
- Stock
ReplicatedMergeTreeacross 3 AZs: Replicas fetch parts peer-to-peer over EC2 TCP port9010. AWS charges$0.02/GB($20/TB) for every byte crossing an AZ boundary:Cross-AZ Tax = 1.974 TB/day × 30.4 days × (4…6.7 cross-AZ hops) × $20/TB = $4,800 to $8,000/month - Antalya CAS across 3 AZs (
multi_az_config.enabled = true): Cold parts are written once to regional S3 via the free S3 Gateway Endpoint ($0/GB), and replicas in other AZs relink the S3 blob viaHEADand read through the regional S3 Gateway Endpoint at$0.00/GBcross-AZ transfer! That is why our Terraform root module enforces a plan-time guard invariables.tf:validation { condition = !var.multi_az_config.enabled || var.cas_config.enabled error_message = "multi_az_config.enabled requires cas_config.enabled = true (Altinity Antalya CAS). Stock ReplicatedMergeTree replicas fetch parts peer-to-peer, so multi-AZ placement bills $0.02/GB cross-AZ on every part fetch (+$4,800–$8,000/mo at 30 TB/day)." }
C. Halving Customer Retention Add-On COGS (Protecting Gross Margin on Low-Compressibility Logs)
When pricing a log analytics SaaS product, every extra 30 days of raw log
retention adds cold S3 cost proportional to r / c:
| Log Payload Profile | Compression Ratio (c) |
Non-CAS Unsharded (r = 10) |
Non-CAS Sharded (r = 2) |
Antalya CAS (r = 1) |
|---|---|---|---|---|
| ALB / NLB / Structured NDJSON | 15.2× |
+$0.0151 / GB |
+$0.0030 / GB |
+$0.0015 / GB |
| Kubernetes / Microservice JSON | 8.0× |
+$0.0288 / GB |
+$0.0058 / GB |
+$0.0029 / GB |
| High-Entropy / UUID / Security Traces | 5.0× |
+$0.0460 / GB |
+$0.0092 / GB |
+$0.0046 / GB |
On high-entropy security logs (c = 5×) retained for 90 days (3 × 30d
windows), non-CAS even at r = 2 costs $0.0276/GB in S3 alone —
whereas CAS (r = 1) cuts that in half to $0.0138/GB.
4.1 Measured vs. Modeled: What the 83.85 TB/day 10-Minute Burst Confirmed (2026-11-18)
After the post above landed, a controlled burst on the internal 30 TB/day validation fleet (2,987 files,
287,257,259 rows, 476,598 rec/s, 10m 03s) produced the following against the model:
| Metric | Modeled in this post | Measured (10-min burst) |
|---|---|---|
| Writers at saturation | 3 × m8g.4xlarge at 60–90% CPU |
92–99% CPU, 13–21 GiB RAM (fits) |
| Reader behavior | “Readers fetch finished parts, never re-merge” | Held only for ≥16 GiB readers. An 8 GiB c8g.xlarge hit Code: 241 with 32 concurrent merges (8 × concurrency 4); fixed by concurrencyRatio = 2 at ≤12 GiB RAM and by requiring ≥16 GiB readers. |
Cold-tier S3 API $/day (CAS) |
$/TB × V × r with r = 1 |
$0.25–0.36/hr ($6–9/day) — consistent with the ~$161/mo figure measured during the 1-hour soak |
cas/ bucket growth during burst |
near-zero (hot parts) | 3.95 GB (unchanged; 41.8 GiB stayed in hot gp3) |
| S3 API spend for entire burst | n/a | $0.0694 total (Tier1=8,005, Tier2=71,125) |
| Staging backlog behavior | “grows then drains” | Drained to zero before the generator stopped; 186 staging objects at burst end |
Two deltas are worth acting on in future post revisions:
- Readers are not free. The model treats readers as negligible in RAM terms; the
burst OOM shows the binding constraint is merge concurrency headroom, not CPU. Size
readers at
≥16 GiBand capconcurrencyRatioby physical RAM, not vCPU. - Burst > soak for cost validation. A 7-day soak at
30 TB/daynever stressed the same code paths as83.85 TB/dayfor 10 minutes (which hit RAM and Keeper DDL-znode paths the steady state never touched). Include burst tests in the model’s validation checklist for future quarterly refreshes.
5. End-to-End 30 TB/day Company COGS: All Four Operating Postures
Putting upstream ingest (Lambda + SQS + 3-day raw S3 backstop = $708/mo),
Graviton4 compute (m8g.2xlarge writers at $262/mo OD / $162/mo 1-yr
SP, c7g.xlarge readers, 3× t4g.large Keepers), EBS, S3 API, network,
and cold S3 together:
Operating Posture (30 TB/day = 912,000 GB/mo) |
Cold S3 Copies (r) |
ClickHouse Cluster $/mo (7d / 33d) | Upstream Ingest $/mo | Total Company COGS $/mo (7d / 33d) | Unit COGS per GB (7d / 33d) |
|---|---|---|---|---|---|
A. Unoptimized Defaults (1-shard r = 10, 45d raw S3, CW info, PUT storm) |
10× |
~$21,017 |
~$5,538 |
~$26,555/mo (+$6.5k if 3-AZ) | 2.91¢ / GB |
| B. Prod GA · Zero-Lag On-Demand (3 shards × 3 replicas, On-Demand) | 3× |
$4,380 / $7,735 |
$708 |
$5,088/mo / $8,443/mo |
0.56¢ / 0.93¢ / GB |
C. Prod GA · Bounded-Lag + 1-yr SP (3 shards × 2 replicas, 3W m8g.2xl + 3R + 3K) |
2× |
$2,035 / $4,271 |
$708 |
$2,743/mo / $4,979/mo |
0.30¢ / 0.55¢ / GB |
D. Target Floor · Antalya CAS + 1-yr SP (metadata_type = cas shared pool, m8g.2xl) |
1× |
$1,647 / $2,765 |
$708 |
$2,355/mo / $3,473/mo |
0.26¢ / 0.38¢ / GB |
6. What It Actually Took to Enable CAS in dev.tfvars (Four Real-World Traps)
In theory, enabling CAS in our Terraform stack looks like a three-line
addition to vars/dev.tfvars:
cas_config = {
enabled = true
prefix = "cas"
}
In practice, when we traced the entire chain from GitHub Actions Packer
builds to S3 bucket lifecycles and bootstrap-ddl.sql, we hit four
real traps that anyone enabling CAS on AWS needs to know.
Trap 1: Building Upstream and Antalya Golden AMIs in Parallel (Without Killing Each Other)
Upstream ClickHouse (26.8.36.3) does not include the metadata_type = cas
disk driver — it lives only in the Apache-2.0 Altinity Antalya build
(26.6.8.20004.altinityantalya). If you flip cas_config.enabled = true
while your nodes still boot an upstream AMI, clickhouse-server fails on
startup when mounting <cas>.
We updated build-golden-ami.yml in stackgenhq/chinit to build both
flavors (upstream and antalya) + chinit in parallel on every release.
Doing that exposed three subtle CI traps:
- GitHub Actions
matrixcontext in job-levelif:: Referencingmatrix.flavorinsidejobs.build-ami.ifcauses GitHub Actions to fail the workflow in0sat parse time (context "matrix" is not allowed here), because job-levelif:evaluates before matrix expansion. Instead, dynamically filterstrategy.matrix.flavorusingfromJSON(...):strategy: fail-fast: false matrix: flavor: >- $ - Packer
-var-fileprecedence: In Packer HCL2, later arguments override earlier ones. Passing-var-file=defaults.pkrvars.hcl(which setsclickhouse_version = "26.8.36.3") after-var clickhouse_version=26.6.8.20004.altinityantalyasilently overwrites the Antalya version with26.8.36.3! Always pass-var-filefirst and explicit-varoverrides after. - Per-flavor
BUILD_IDandFlavortag isolation: Our workflow has anif: always()cleanup step that sweeps any EC2 builder instance, key pair, or security group tagged withBuildId. Whenupstreamandantalyarun in parallel, sharingBUILD_ID: $-$means whichever job finishes first terminates the other job’s running Packer builder! ScopingBUILD_IDto$-$-$and stampingFlavor = var.build_flavoron the AMI tags lets both builders run concurrently and retainKEEP_GENERATIONS = 3per flavor.
Trap 2: S3 Lifecycle Rules Scoped Only to data/ Miss cas/
In modules/storage/main.tf, our data bucket lifecycle rule
(tier-cold-parts-to-ia) was filtered on prefix = "data/".
When CAS is enabled, cold blobs are written under cas/tiered/ and
cas/<tenant>/ — outside data/. And as covered in the
CAS staging walkthrough, a
7-day AbortIncompleteMultipartUpload rule on the CAS prefix is a
mandatory precondition so interrupted multipart uploads from spot/ASG
churn never accumulate orphan fragments. Adding a dynamic tier-cas-parts-to-ia
lifecycle rule scoped to local.cas_prefix_normalized (cas/) closed the
gap.
Trap 3: Doubling Cache Wrappers Halves Local Cache Size (32 GiB → 64 GiB)
In modules/clickhouse/locals.tf, total local cache disk capacity
(0.9 × cache_disk_size_gb) is divided across all configured cache
wrappers:
cache_wrapper_count = 1 + length(var.tenants) + (var.cas_enabled ? length(var.tenants) + 1 : 0)
With 1 tenant (default):
- CAS off:
cache_wrapper_count = 2(s3_cache+cache_default). On a32 GiBcache volume, each wrapper gets⌊32 × 0.9 / 2⌋ =14 GiB. - CAS on:
cache_wrapper_count = 4(s3_cache,cache_default,cas_cache,cas_cache_default). On the same32 GiBvolume, each wrapper drops to⌊32 × 0.9 / 4⌋ =7 GiB!
Bumping instance_config.cache_disk_size_gb from 32 to 64 in
dev.tfvars costs +$2.56/month per node (32 GiB × $0.08/GiB-mo)
and restores the full 14 GiB per cache wrapper.
Trap 4: Table-Level DDL Overrides (storage_policy and execute_merges_on_single_replica_time_threshold)
Turning on cas_config.enabled = true renders the tiered_cas policy
into storage.xml and sets merge_tree/execute_merges_on_single_replica_time_threshold: "3600"
in chi-spec.yaml. Neither takes effect on default.events if your
bootstrap DDL hardcodes the non-CAS values:
bootstrap-ddl.sqlhardcodedSETTINGS storage_policy = 'tiered'. And ClickHouse’sStoragePolicy::checkCompatibleWithrejectsALTER TABLE default.events MODIFY SETTING storage_policy = 'tiered_cas'on an existing table becausetiered_casdoes not include the olds3_cachedisk.bootstrap-ddl.sqlhardcodedexecute_merges_on_single_replica_time_threshold = 0andremote_fs_execute_merges_on_single_replica_time_threshold = 0in theSETTINGSclause ofdefault.eventsand the three summary tables — silently overriding the server-level3600inchi-spec.yaml!
Parameterizing bootstrap-ddl.sql from modules/clickhouse/locals.tf so
storage_policy renders ${storage_policy} (tiered_cas when
cas_enabled = true) and both CREATE TABLE and the idempotent
ALTER TABLE ... MODIFY SETTING convergence block render
${single_replica_merge_threshold} (3600 when cas_enabled = true, 0
otherwise) ensures new tables and converged settings align automatically.
7. Honest Trade-Offs: When to Use CAS vs. When Sharding Is Enough
CAS is the most powerful storage cost lever in open-source ClickHouse, but it is not a free lunch:
| Dimension | Sharded Stock ClickHouse (26.8 LTS, S = 3, r = 2) |
Altinity Antalya CAS (26.6.8+, r = 1) |
|---|---|---|
Cold S3 Copies (r) |
2× ($636/mo @ 7d · $2,872/mo @ 33d) |
1× ($318/mo @ 7d · $1,436/mo @ 33d) |
Single-Shard High Replica Count (S = 1, R = 10) |
Expensive (10× S3 = $14,359/mo @ 33d) |
Cheap (1× S3 = $1,436/mo @ 33d) |
| Multi-AZ Compute Placement | Prohibitive (+$4,800–$8,000/mo cross-AZ TCP) |
Viable (~$15–$40/mo via free S3 Gateway Endpoint) |
| Build Maturity | Upstream LTS GA | Experimental (Apache-2.0 Altinity Antalya fork; pin exact versions) |
| Cold-Tier Blast Radius | Per-replica (data/tiered/writer-0/ loss affects 1 node) |
Shared pool (cas/tiered/ corruption affects all replicas; nightly clickhouse-backup to a separate versioned/object-locked bucket is mandatory) |
Summary Checklist for Anyone Running ClickHouse on AWS
- Lock down the S3 Gateway Endpoint first: Attach
com.amazonaws.<region>.s3to every private route table (ClickHouse nodes, EKS pods, and VPC-attached Lambdas), pass--regionon everyaws s3CLI call inuser-data, and enforce same-region bucket checks attofu plantime. - Fix part shape before blaming S3 pricing: Use
256+ GiBhotgp3volumes,min_bytes_for_wide_part = 1073741824(1 GiB),ttl_only_drop_parts = 1, and a shared S3Queuekeeper_pathwithenable_hash_ring_filtering = 1. - Eliminate the
R×cold-storage multiplier: Either shard the cluster (S = 3, r = 2) on stock ClickHouse LTS, or deploy the Altinity Antalya Golden AMI withcas_config.enabled = trueto get a true1×content-addressed S3 cold pool and0.26¢–0.38¢/GBend-to-end unit economics.
FAQ
If we use an S3 Gateway Endpoint, why does S3 still cost thousands of dollars a month?
An S3 Gateway Endpoint (com.amazonaws.
What silent bugs cause S3 traffic to bypass an S3 Gateway Endpoint?
Four things: (1) calling aws s3 cp in cloud-init user-data without --region or AWS_REGION set, which defaults to s3.amazonaws.com (us-east-1) and exits via NAT because the us-west-2 endpoint prefix list only matches us-west-2 S3 IPs; (2) leaving region blank in clickhouse-backup's config.yml when node identity metadata is empty; (3) deploying a secondary EKS or VPC-attached Lambda subnet whose private route table was never attached to the gateway endpoint; and (4) pointing S3Queue at a bucket in another region.
How much S3 storage money does CAS actually save at 30 TB/day?
At 30 TB/day of uncompressed logs with 15.2x ZSTD compression (1.974 TB/day compressed), one cold copy costs $318/month at 7-day retention and $1,436/month at 33-day retention. On a 1-shard cluster with a 10-replica autoscale ceiling, stock ClickHouse stores 10 physical copies ($3,178/mo at 7d; $14,359/mo at 33d). CAS collapses that to 1 shared content-addressed pool ($318/mo at 7d; $1,436/mo at 33d) — saving $2,860/month at 7d and $12,923/month ($155k/year) at 33d. Even against a 3-shard cluster with 2-3 replicas per shard, CAS cuts cold S3 spend by 50% to 67% ($1,436-$2,872/month saved at 33d).
How does CAS eliminate the $4,800-$8,000/month Multi-AZ replication tax?
Stock ReplicatedMergeTree fetches parts peer-to-peer between EC2 instances over TCP port 9010, which AWS bills at $0.02/GB across Availability Zones. With CAS, a writer uploads a cold part blob once to regional S3 ($0/GB via the S3 Gateway Endpoint), and replicas in other AZs relink the blob by its content hash (HEAD + manifest reference) and read through regional S3 at $0/GB cross-AZ transfer.
What happens if you just set cas_config.enabled = true in Terraform without changing anything else?
Four things go wrong if you only flip the flag: (1) upstream ClickHouse binaries do not have metadata_type = cas and refuse to boot without an Altinity Antalya Golden AMI; (2) existing tables stay on storage_policy = 'tiered' because ClickHouse forbids in-place policy swaps that drop old disks; (3) table-level execute_merges_on_single_replica_time_threshold = 0 in DDL overrides the server-level 3600s setting; and (4) adding cas_cache and cas_cache_
Stay in the loop — production notes on AI agents, workflows, and SRE.
Low volume — new posts and curated reading lists. Unsubscribe anytime.