Where does the money go on an MSK provisioned cluster?
On a provisioned MSK cluster, the bill comes from three main sources, and each one grows with a different property of your workload.
| Cost driver | What you pay for | Scales with |
|---|---|---|
| Broker-hours | Instance type × broker count × hours running | Peak throughput, partition count, CPU/network headroom |
| Provisioned EBS storage | GB-months allocated per broker, used or not | Retention × ingest rate × replication factor |
| Data transfer | Client traffic crossing AZ or region boundaries | Producer/consumer placement and fan-out |
Storage hides a multiplier. With a replication factor of 3, every byte a producer writes is stored on three brokers' local EBS volumes. Seven days of retention at a given ingest rate is billed as twenty-one days of single-copy storage. You also pay for provisioned capacity, not used capacity. Because a full broker disk is an outage, teams keep free headroom on every volume, and that idle space is billed too. Extending retention therefore raises storage cost by roughly RF times the new data, plus the headroom you add to stay safe.
This article sorts every cost by one distinction:
- Bytes-in-flight are data being produced, replicated, and consumed right now. They determine broker size, network, and transfer cost, and they scale with throughput.
- Bytes-at-rest are data kept only because retention says so. Most of it is rarely read, yet it sits on replicated local disk. It scales with retention.
Tiered storage only addresses bytes-at-rest. If your bill is dominated by broker-hours or cross-AZ traffic, tiering will not change much. If it is dominated by EBS, tiering can reduce cost substantially.
What is MSK tiered storage and how does it work?
Kafka stores each partition as a sequence of log segments. Only the newest, the active segment, receives writes. Once a segment rolls and closes, it is immutable, so it can be copied elsewhere without coordination with producers. Tiered storage relies on this. Closed segments are uploaded to a remote tier that AWS manages, and the brokers delete their local copies once local retention expires. The active segment and recent closed segments stay on broker EBS, where they serve producers and caught-up consumers at normal latency.
The billing difference comes from replication. On EBS, each byte is stored once per replica, so a topic with replication factor 3 pays for EBS three times. The remote tier is billed per GB stored, and that charge is not multiplied by replication factor, because durability is handled by the storage layer instead of by Kafka replicas.
Consumers keep using the same API. When a consumer fetches an offset that has been moved out of local storage, the broker reads it from the remote tier. These reads have higher latency than local reads, which matters for backfills and replays.
Two sets of settings control the behavior:
- Total retention (
retention.ms/retention.bytes) sets how long data exists anywhere, local or remote. - Local retention (
local.retention.ms/local.retention.bytes) sets how long data stays on broker EBS before only the remote copy remains.
The savings come from the gap between these two values. Suppose total retention is 7 days and local retention is 6 days. Nearly all the data still sits on EBS at full replication cost, and you also pay for the remote copy. Setting local retention to hours instead moves most of the stored bytes off EBS.
Check the current AWS documentation before planning a rollout. It covers supported Kafka versions and instance types, whether compacted topics can be tiered, whether tiering can be turned off after it is enabled, and any charges beyond per-GB storage.
# Placeholders: replace with your MSK bootstrap brokers and client auth properties file.
BOOTSTRAP="b-1.example-cluster.abc123.kafka.us-east-1.amazonaws.com:9098"
CLIENT_PROPS="client.properties"
# Create a topic with tiered storage enabled.
# retention.ms = total retention, local + remote (7 days)
# local.retention.ms = time data stays on broker EBS (6 hours)
# The savings come from the gap: only the last ~6h are stored on EBS
# once per replica. The rest exists only in the remote tier.
kafka-topics.sh --bootstrap-server "$BOOTSTRAP" \
--command-config "$CLIENT_PROPS" \
--create --topic orders-events \
--partitions 12 --replication-factor 3 \
--config remote.storage.enable=true \
--config retention.ms=604800000 \
--config local.retention.ms=21600000
# Anti-pattern: local retention close to total retention (6 of 7 days).
# Almost every byte stays on EBS at full replication cost, and the
# remote copy is billed on top. Avoid this:
# --config local.retention.ms=518400000
# Inspect the effective settings for the topic.
kafka-configs.sh --bootstrap-server "$BOOTSTRAP" \
--command-config "$CLIENT_PROPS" \
--describe --entity-type topics --entity-name orders-events
Creates an MSK topic with tiered storage enabled. It keeps 7 days of total retention but only 6 hours on broker EBS, and it shows the near-equal local retention setting that cancels most of the savings.
How much can tiered storage save? A worked illustrative estimate
Illustrative estimate (assumed numbers, not a benchmark). Assume 50 MB/s average ingest (≈4.32 TB/day), replication factor 3, and a headroom factor of 1.4 so disks stay around 70% full. Required local EBS is:
local_TB = daily_ingest × local_days × RF × headroom
For the tiered layout, assume one day stays local and the full 30 days live in the remote tier. The remote tier holds one logical copy rather than RF copies. Confirm how your rate card meters it.
| Layout | Local EBS (TB) | Remote (TB) | Monthly storage cost |
|---|---|---|---|
| All-local, 7 days | 4.32 × 7 × 3 × 1.4 ≈ 127 | 0 | 127 · P_ebs |
| All-local, 30 days | 4.32 × 30 × 3 × 1.4 ≈ 544 | 0 | 544 · P_ebs |
| Tiered, 1 day local / 30 days total | 4.32 × 1 × 3 × 1.4 ≈ 18 | 4.32 × 30 ≈ 130 | 18 · P_ebs + 130 · P_remote + R · P_retrieval |
Compared with 30-day all-local storage, tiering breaks even when:
526 · P_ebs > 130 · P_remote + R · P_retrieval
R is the volume read back from the remote tier each month. The inequality favors tiering because it removes both the RF multiplier and the headroom multiplier from 29 of the 30 days.
P_broker belongs in the comparison only if storage, not throughput, sets your broker count. If storage limits are what force extra brokers, shrinking local disk can remove brokers and add (Δbrokers × P_broker) to the savings. If throughput sets the broker count, that term cancels out.
Retrieval is where the savings can turn into a shifted cost. Suppose a replay or backfill job rereads the full 30 days three times a month. That makes R ≈ 390 TB, and at a non-trivial P_retrieval this can consume much of the 526 · P_ebs margin. Re-read-heavy workloads should model R from observed consumer lag and replay patterns before committing to tiered storage.
# Illustrative estimate only: every input below is an assumption, not a measurement.
# Replace the P_* placeholders with current rates from your own rate card.
INGEST_MB_S = 50 # assumed average ingest
RF = 3 # replication factor on local EBS
HEADROOM = 1.4 # keeps disks around 70% full
TOTAL_DAYS = 30 # total retention
LOCAL_DAYS = 1 # local window kept on EBS when tiered
P_ebs = 1.0 # placeholder: cost per TB-month of EBS
P_remote = 1.0 # placeholder: cost per TB-month in the remote tier
P_retrieval = 1.0 # placeholder: cost per TB read back from remote
P_broker = 1.0 # placeholder: cost per broker-month
daily_tb = INGEST_MB_S * 86_400 / 1_000_000 # ~4.32 TB/day (decimal units)
def local_tb(days):
# Local disk pays both the RF and the headroom multipliers.
return daily_tb * days * RF * HEADROOM
all_local = local_tb(TOTAL_DAYS) # ~544 TB
tiered_local = local_tb(LOCAL_DAYS) # ~18 TB
remote_tb = daily_tb * TOTAL_DAYS # ~130 TB, one logical copy
def tiered_cost(reread_tb, brokers_removed=0):
# brokers_removed > 0 only if storage, not throughput, sets broker count.
return (tiered_local * P_ebs + remote_tb * P_remote
+ reread_tb * P_retrieval - brokers_removed * P_broker)
margin = (all_local - tiered_local) * P_ebs # the ~526 * P_ebs side
for replays in (0, 1, 3): # full 30-day rereads per month
R = replays * remote_tb # 3 replays -> ~390 TB
cost = remote_tb * P_remote + R * P_retrieval
print(f"replays={replays} R={R:.0f}TB "
f"tiering_wins={margin > cost} "
f"tiered={tiered_cost(R):.1f} all_local={all_local * P_ebs:.1f}")
A small Python model of the illustrative estimate that computes local and remote TB from the assumed inputs, keeps every price as a placeholder variable, and checks the break-even inequality as replay-driven retrieval volume R increases.
Who should use tiered storage, and when is it the right tool?
Tiering fits when three conditions hold together.
- Retention is driven by replay, audit, or reprocessing. If topics must hold weeks or months of data so a team can rebuild a downstream store or satisfy an auditor, most of those bytes sit idle. Idle bytes are what the remote tier is priced for.
- Consumers read near the head of the log. Tiering only saves money if hot reads stay on local disk. Cold fetches add latency and, as the previous section showed, can add retrieval cost.
- Disk is the constraint that forces broker additions. If you add brokers because of CPU, network throughput, or partition count, tiering reduces your storage line but not your broker count. Most of the savings disappear.
Measure before deciding
Start with per-topic size on disk. Sum the broker Log Size metric across partitions and compare it with KafkaDataLogsDiskUsed in CloudWatch. This shows which topics drive disk growth. Usually a few topics account for most of it.
Next, look at the consumer lag distribution in time, not offsets. MSK's EstimatedMaxTimeLag per consumer group and topic gives this directly. Any group whose time lag regularly exceeds your planned local.retention.ms will read from the remote tier.
Finally, estimate the share of fetch traffic that targets old offsets. Weight each consumer group's fetch bytes by whether its lag falls outside the local window. Add scheduled replays and backfills, which lag metrics miss because they start from early offsets.
That weighted share is your predicted remote-hit fraction, and it is the input for R. If it is near zero, tiering is cheap. If it is large, you need a longer local window or a different approach.
# Estimate the remote-hit fraction (R) from measured inputs.
# Values below are placeholders (illustrative estimate), not measurements.
# fetch_bytes_per_day: per-group fetch volume for the topic.
# p95_time_lag_s: from EstimatedMaxTimeLag (seconds), taken over a typical week.
LOCAL_RETENTION_MS = 6 * 60 * 60 * 1000 # planned local.retention.ms (6 hours)
local_window_s = LOCAL_RETENTION_MS / 1000
consumer_groups = [
# (group, fetch_bytes_per_day, p95_time_lag_s)
("orders-enricher", 400e9, 30),
("search-indexer", 250e9, 900),
("nightly-analytics", 120e9, 30_000), # batch job, lags past the local window
]
# Scheduled replays/backfills start from early offsets, so lag metrics miss them.
# Count their expected bytes as remote reads, averaged per day.
replay_bytes_per_day = 2 * 1.5e12 / 30 # e.g. two 1.5 TB replays per month
local_bytes = 0.0
remote_bytes = replay_bytes_per_day
for group, fetch_bytes, lag_s in consumer_groups:
# A group whose lag exceeds the local window reads from the remote tier.
if lag_s > local_window_s:
remote_bytes += fetch_bytes
else:
local_bytes += fetch_bytes
total = local_bytes + remote_bytes
r = remote_bytes / total
print(f"remote bytes/day: {remote_bytes / 1e9:.0f} GB")
print(f"predicted remote-hit fraction R: {r:.1%}")
# R near zero: tiering is cheap. Large R: lengthen local.retention.ms
# or reconsider whether tiering fits this topic.
An illustrative estimate of how often the remote tier will be hit. It weights each consumer group's fetch bytes by whether its time lag exceeds the planned local.retention.ms, and it adds scheduled replays that lag metrics do not capture.
When should you avoid MSK tiered storage?
Tiering is the wrong tool when the remote-hit fraction stays high no matter how you tune the local window, or when the cluster's cost isn't driven by storage at all.
Remote reads are slower and less predictable
Fetches that miss local disk go to object storage. Latency is higher and more variable than reading from a broker's page cache or local volume. Routine tailing consumers never notice. Three workloads do:
- Backfills
- New consumer groups starting from earliest
- Incident replays
These jobs take longer, and heavy remote fetching consumes broker network and threads that serve normal traffic. Remote retrieval is also billed separately from tiered storage, so a replay-heavy workload pays twice: once in time and once in fees.
Little gain on throughput- or partition-bound clusters
If you sized brokers for ingress, CPU, or partition count rather than disk, moving bytes off-broker doesn't let you drop instances. The storage line shrinks slightly while the compute line, the bigger one, stays where it was.
Operational coupling and failure behavior
After you enable tiering, later changes to retention, disabling tiering, and migrating to another cluster all have to account for data that lives in two places. Broker replacement and partition reassignment also behave differently, because less data is copied locally. Test these paths before relying on them.
The sharpest risk is an aggressively short local window. During an outage, consumers fall behind and drop onto remote reads. Recovery slows at the moment you most need it to be fast.
What are the alternatives to tiered storage for cutting Kafka retention cost?
Before changing storage tiers, clean up retention. Every byte you stop keeping saves money on local and remote storage alike, so hygiene pays off whether or not you tier later.
Start with an audit. Many topics run with broker defaults that nobody chose deliberately. Look for long retention.ms values on debug or staging topics, and check retention.bytes closely. That setting applies per partition, so a cap that looks reasonable on a 48-partition topic allows 48 times that much data per replica.
Next, look at low-value topics with too many replicas or partitions. Replication factor multiplies stored bytes, so a throwaway metrics topic rarely needs the same durability as an orders topic. You can lower replication by reassigning partitions. You cannot shrink a topic's partition count in place, so fixing over-partitioning means creating a new topic and migrating to it.
Two settings cut bytes without changing what data you keep:
- Compaction (
cleanup.policy=compact) keeps only the latest record per key. It suits topics where consumers only need current state. - Producer-side compression (
compression.typeset tolz4orzstd, for example) shrinks data on the wire and on disk.
Plain retention reduction is the cheapest and simplest lever, but it deletes data. Whether you can give up replay depth or audit history is a business decision, not an engineering optimization.
| Approach | Cost reduction | Effort | Risk | Data loss |
|---|---|---|---|---|
| Hygiene (audit, RF, compaction, compression) | Moderate, applies to every tier | Low to moderate, done topic by topic | Low | None if compaction fits the access pattern |
| Shorter retention | High, immediate | Very low, a config change | Lost replay and audit history | Yes, by design |
| Tiered storage | High for long retention | Moderate to high | Slower remote reads during recovery | None |
Hygiene also saves engineering time. It needs no client testing, no capacity remodeling, and no migration plan. Once it is done, you can see more clearly whether you still need tiered storage.
#!/usr/bin/env bash
# Retention hygiene pass for one topic. Placeholders: adjust broker, auth file, topic.
set -euo pipefail
BOOTSTRAP="b-1.example-cluster.kafka.us-east-1.amazonaws.com:9098"
CLIENT_CONF="client.properties" # TLS/IAM settings for your cluster
TOPIC="staging.debug-events"
# 1. Audit: show configs set explicitly on the topic.
# Anything missing here is inherited from broker defaults nobody chose.
kafka-configs.sh --bootstrap-server "$BOOTSTRAP" --command-config "$CLIENT_CONF" \
--entity-type topics --entity-name "$TOPIC" --describe
# 2. Check partition count and replication factor.
# retention.bytes is per partition, so multiply it by partitions x RF
# to get the real worst-case footprint.
kafka-topics.sh --bootstrap-server "$BOOTSTRAP" --command-config "$CLIENT_CONF" \
--describe --topic "$TOPIC"
# 3. Shorter retention: 3 days and a 1 GiB per-partition cap.
# This DELETES older data; confirm replay/audit needs first.
kafka-configs.sh --bootstrap-server "$BOOTSTRAP" --command-config "$CLIENT_CONF" \
--entity-type topics --entity-name "$TOPIC" --alter \
--add-config retention.ms=259200000,retention.bytes=1073741824
# 4. Compaction for a state topic where consumers only need the latest value per key.
kafka-configs.sh --bootstrap-server "$BOOTSTRAP" --command-config "$CLIENT_CONF" \
--entity-type topics --entity-name "inventory.current-state" --alter \
--add-config cleanup.policy=compact
# 5. Compression is set on the producer, e.g. in producer.properties:
# compression.type=zstd
# Leave the topic's compression.type at its default (producer) so brokers
# store batches as the producer compressed them.
Shell commands that audit a topic's explicit configs and partition layout, then apply shorter retention, compaction, and a producer-side compression setting.
How do you implement tiering and actually realize the savings?
Step-by-step rollout
-
Baseline first. Record per-topic disk usage, broker disk utilization, peak consumer lag (in time, not only offsets), and the monthly bill split into broker-hours, provisioned storage, and data transfer. Without this, you cannot tell whether a later drop came from tiering or from a quiet month.
-
Apply retention hygiene. Trim oversized
retention.msvalues and delete dead topics before tiering anything. Otherwise you pay to move data to the remote tier that you should have deleted. -
Enable tiering on selected topics. Set
remote.storage.enable=trueon a few large, append-heavy topics. Setlocal.retention.mswith a margin above your worst observed consumer lag, for example twice the longest lag seen during incidents. Consumers that fall behind local retention read from the remote tier, which is slower. The margin keeps normal catch-up on local disk. -
Test failure and replay. Replace a broker, force a consumer to rewind past local retention, and measure fetch latency and throughput. Confirm that recovery time still fits your SLOs.
-
Right-size. Savings appear only when provisioned EBS or broker count actually goes down. MSK EBS volumes can grow but cannot shrink in place, so reducing local storage usually means building a smaller cluster and migrating to it, for example with MirrorMaker 2. Until the old cluster is gone, you pay for both tiers.
-
Verify and budget. Compare a full billing cycle against the baseline. Estimate payback as:
payback (months) = migration engineering cost ÷ (old storage and broker cost − new storage and broker cost − tiered storage cost)
If the denominator is small or negative, stop at hygiene.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to msk tiered storage cost, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.