How does a replication slot cause unbounded WAL growth?
A replication slot is a promise the primary makes to a consumer: no WAL the consumer still needs will be removed until it confirms receipt. Without a slot, a subscriber that falls behind can find its next segment already recycled and must be rebuilt. The slot prevents that by recording the consumer's position and pinning WAL from that point forward.
Two positions matter. restart_lsn is the oldest WAL the slot may still need. For a logical slot, it can sit behind the consumer's progress because decoding has to restart from the oldest transaction that was still open. confirmed_flush_lsn is what the logical consumer has acknowledged as durably applied.
Normally a checkpoint recycles segments older than its redo point, and wal_keep_size sets a minimum amount of WAL to keep. A slot overrides both. Any segment at or after the oldest restart_lsn across all slots stays in pg_wal, however old it is.
Illustrative layout of pg_wal with a lagging consumer
| Segments | State | Reason |
|---|---|---|
| Oldest kept segment | Retained | Contains the slot's restart_lsn |
| Middle segments | Retained | Past confirmed_flush_lsn, but still behind the slot's horizon |
| Newest segment | Being written | The primary's current insert position |
The guarantee has no time limit. If the consumer stalls, the slot's positions freeze, and every new write adds WAL that cannot be removed. A stalled consumer might be a crashed CDC connector, a paused subscription, or a slot left behind after a migration. Retained WAL grows with write volume until the WAL volume is full. At that point the primary cannot write WAL, so it stops accepting writes and typically shuts down until space is freed.
+---------+
| Clients |
+----+----+
| writes
v
+----------------------------------------------------------------+
| PRIMARY pg_wal/ on WAL volume |
| |
| +------+------+------+------+------+------+------+ |
| | seg1 | seg2 | seg3 | seg4 | seg5 | seg6 | seg7 | ---> grows |
| +------+------+------+------+------+------+------+ |
| ^ ^ ^ |
| | | | |
| restart_lsn confirmed_flush_lsn insert position |
| <------- RETAINED, never recycled --------> |
| (checkpoints and wal_keep_size cannot override the slot) |
| |
| replication slot (consumer stalled -> positions frozen) |
| WAL volume full -> no new WAL -> writes stop |
+-----------------+----------------------------------------------+
|
| decoded changes, acks of confirmed_flush_lsn
X consumer stopped: no reads, no acks
|
+---------+---------+
| Lagging consumer |
| (crashed, paused, |
| or abandoned) |
+-------------------+
The slot's restart_lsn and confirmed_flush_lsn stay frozen while the consumer is stalled, so pg_wal keeps every segment from restart_lsn up to the current insert position until the WAL volume fills and the primary can no longer accept writes.
Why are logical replication slots riskier than physical ones?
Physical slots usually feed a standby that the database team runs, monitors and can rebuild. Logical slots feed change data capture (CDC) pipelines, cross-database replication and event streaming connectors. Those consumers are often owned by another team, deployed on their own schedule, and not covered by the database on-call rotation. When one stops consuming, the primary keeps retaining WAL, and the people paged for the disk alert may not be able to fix the consumer.
Logical consumers also stall for more reasons:
- Schema changes. Logical replication does not carry DDL. A column added on the publisher but missing on the subscriber makes apply fail, and the subscriber retries indefinitely.
- Decoding and apply errors. Output plugin failures, constraint violations and type mismatches all stop progress at a single change.
- Downstream backpressure. If a message broker or sink is slow, the connector stops confirming positions.
- Orphaned slots. A deleted connector, or a subscription dropped without dropping its slot, leaves a slot that nothing will ever read from.
Decoding itself holds WAL back. A long-running transaction pins restart_lsn, because decoding must restart from that transaction's start. Very large transactions can spill to disk on the primary while they are decoded. On an idle publisher, some connectors only advance confirmed_flush_lsn when they see a change, so writes to other databases accumulate WAL against the slot. Teams often handle this with a heartbeat table.
| Risk factor | Physical slot | Logical slot |
|---|---|---|
| Typical consumer owner | Database team | Often another team |
| Stalls on DDL | No | Yes |
| Long transactions pin WAL | No | Yes |
| Can advance while idle | Yes, through replay | Depends on the consumer |
Catalog bloat (catalog_xmin) |
No | Yes |
Who should use these controls?
Any team running a PostgreSQL primary that has at least one replication slot should monitor slot retention and decide on a cap. Running SELECT count(*) FROM pg_replication_slots; on the primary tells you whether this applies to you. Slots are often created by tooling rather than by hand, so the answer may be yes even if nobody on the team remembers creating one.
The controls in this article matter most for:
- Teams with logical replication or CDC. Subscriptions, event streaming connectors and data pipeline tools usually create logical slots, and those consumers stall for more reasons than a standby does.
- Platform and DBA teams running shared clusters. When many applications share one primary, a single stalled consumer owned by one team can stop writes for all of them. These teams need per-slot alerts and a documented policy on caps and dropping slots.
- Teams using physical standbys with slots. The risk is lower because the database team usually owns the standby, but a standby that is down for a long time still pins WAL.
- Users of managed PostgreSQL services. You may not control the disk directly, but slot retention still consumes your storage, and the parameters and queries here generally still apply, subject to your provider's documentation.
If a cluster has no slots today, the controls still belong in the checklist for whoever adds the first one, because that is when the risk starts.
How do you read pg_replication_slots and measure retained WAL?
Each row in pg_replication_slots is one slot, and a handful of columns tell you whether it is pinning WAL:
slot_name: the identifier the consumer uses. Name slots after the consuming service so ownership is obvious during an incident.slot_type:physicalorlogical.active: whether a consumer is currently connected. A slot withactive = falsestill retains WAL; disconnecting does not release anything.restart_lsn: the oldest WAL position the slot may still need. The server cannot remove WAL at or after this point, so it is the column that drives disk usage.confirmed_flush_lsn: for logical slots, the position the consumer has acknowledged. It can sit well ahead ofrestart_lsnbecause decoding has to start at the beginning of the oldest open transaction.wal_status: on versions that have it, this shows whether the slot's required WAL is still within normal retention or at risk of being removed. A value oflostmeans the slot can no longer be used.
To measure retained WAL, run this on the primary for each slot: pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)). Sort by the raw byte value, not the pretty string, so the worst offender appears first. For inactive slots, filter on WHERE NOT active and include the same size expression. An idle slot holding gigabytes is usually an abandoned consumer.
Columns and their meanings have changed across PostgreSQL releases. Check the documentation for your version before you build alerts on any of them.
-- Retained WAL per slot, worst offender first.
-- Run on the primary. Requires PostgreSQL 10+ for pg_current_wal_lsn();
-- the wal_status column requires PostgreSQL 13+ (remove it on older versions).
SELECT
slot_name,
slot_type,
active,
restart_lsn,
confirmed_flush_lsn, -- NULL for physical slots
wal_status,
pg_size_pretty(
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
) AS retained_wal
FROM pg_replication_slots
-- Sort on the raw byte count, not the pretty string.
-- NULLS LAST keeps slots with no restart_lsn (never used, or lost) at the bottom.
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC NULLS LAST;
-- Inactive slots still pin WAL. An idle slot holding gigabytes
-- usually means an abandoned consumer.
SELECT
slot_name,
slot_type,
database, -- NULL for physical slots
restart_lsn,
pg_size_pretty(
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
) AS retained_wal
FROM pg_replication_slots
WHERE NOT active
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC NULLS LAST;
Two queries for the primary. The first reports how much WAL each replication slot retains, ordered by raw bytes with the largest first. The second lists only inactive slots so you can find abandoned consumers.
What should you monitor and alert on before the disk is full?
A disk-usage alert at 85% tells you that something is wrong but not what, and it fires late. Normal WAL churn, table bloat, and log files all share that volume, so by the time the threshold trips, much of the headroom is already gone and on-call still has to find the cause. Alert on the cause instead, per slot. Track retained bytes, the distance from the current WAL position to the slot's restart_lsn, and inactivity, meaning a slot whose consumer has been disconnected longer than you expect.
Illustrative estimate
Assume a WAL volume with 200 GB free during normal operation and peak WAL generation of 25 GB/hour. If a consumer stops, retained WAL grows at roughly that rate, so time-to-full is 200 / 25 = 8 hours. Suppose you want at least 4 hours for someone to diagnose and act. You might then:
- Warn at 50 GB retained (6 hours remaining at peak).
- Page at 100 GB retained (4 hours remaining).
- Warn on any slot inactive for more than 15–30 minutes, before retention becomes large.
Base these thresholds on your measured peak rate, not the average. Overnight batch jobs can multiply the rate.
Trade-offs
Per-slot alerts need upkeep. When slots are created or renamed, the alert rules must follow, so generate them from the slot list rather than hand-writing each one. Bursty consumers, such as nightly CDC catch-up, can trip byte thresholds legitimately. Give those slots their own thresholds or a short sustain window instead of loosening the alert for everyone.
The payoff is that the page names the offending slot. On-call skips the disk forensics and starts from the specific consumer that stopped advancing.
-- Per-slot WAL retention and inactivity check. Run on the primary
-- (pg_current_wal_lsn() errors during recovery).
-- Feed rows into your alerting system so each alert names the slot.
-- Thresholds match the illustrative estimate: 25 GB/h peak, 200 GB free.
SELECT
slot_name,
slot_type,
database,
active,
active_pid,
-- Bytes of WAL this slot forces the server to keep
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_bytes,
pg_size_pretty(
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
) AS retained_pretty,
wal_status, -- 'extended' or 'unreserved' means trouble is near
-- inactive_since exists in PostgreSQL 17+. On older versions,
-- track first-seen-inactive time in your monitoring agent instead.
now() - inactive_since AS inactive_for,
CASE
WHEN pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
>= 100 * 1024^3 THEN 'page' -- ~4 h left at peak
WHEN pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
>= 50 * 1024^3 THEN 'warn' -- ~6 h left at peak
WHEN NOT active
AND now() - inactive_since > interval '20 minutes'
THEN 'warn' -- consumer gone, act early
ELSE 'ok'
END AS severity
FROM pg_replication_slots
-- Slots that never reserved WAL have NULL restart_lsn; skip them
WHERE restart_lsn IS NOT NULL
ORDER BY retained_bytes DESC;
A query over pg_replication_slots that reports retained WAL bytes and inactivity per slot, with severity levels taken from the illustrative 50 GB warn and 100 GB page thresholds. inactive_since requires PostgreSQL 17 or later.
When should you set max_slot_wal_keep_size, and how do you size it?
max_slot_wal_keep_size caps how much WAL any single slot can force the server to retain. The default, -1, means no limit. With a cap set, a slot that falls further behind than the cap loses its claim on old WAL at the next checkpoint. The server removes those segments and invalidates the slot. A logical consumer cannot recover from that. You must drop and recreate the slot and resynchronize the subscriber's data, which for large tables can take hours.
wal_status in pg_replication_slots shows where a slot stands:
reservedandextendedare normal states.unreservedmeans the slot is past the cap but its segments still exist, so it can recover if it catches up before the next checkpoint.lostmeans the slot has been invalidated.
safe_wal_size shows how many bytes remain before a slot becomes unreserved. Alert on that value shrinking.
Size the cap from disk capacity, not from how much consumer lag you want to allow. Here is an illustrative estimate using assumed inputs: a 500 GB WAL volume, with roughly 100 GB reserved for normal WAL, max_wal_size, and checkpoint spikes, and 50 GB as a safety margin, leaves a cap of about 350 GB. The cap applies per slot, so with several slots, divide the budget accordingly.
Avoid setting the cap too low. Planned maintenance or a brief consumer outage would then invalidate slots that would have caught up on their own. Setting it too high means the cap only acts after the disk is already in danger.
To apply it, set max_slot_wal_keep_size = '350GB' and run SELECT pg_reload_conf();. Then confirm the change with SHOW max_slot_wal_keep_size;. Check your version's documentation and your managed service's documentation to confirm the parameter exists there and behaves as described.
-- Requires PostgreSQL 13 or later. On managed services, set this through the
-- provider's parameter mechanism instead of ALTER SYSTEM.
-- Cap the WAL any single slot can retain (illustrative value from the
-- disk-budget estimate: 500 GB volume - 100 GB normal WAL - 50 GB margin).
ALTER SYSTEM SET max_slot_wal_keep_size = '350GB';
-- The parameter takes effect on reload; no restart needed.
SELECT pg_reload_conf();
-- Confirm the running value.
SHOW max_slot_wal_keep_size;
-- Check each slot's standing against the cap.
-- safe_wal_size: bytes left before the slot becomes 'unreserved'
-- (NULL when the cap is -1 or the slot is already lost).
SELECT slot_name,
slot_type,
active,
wal_status, -- reserved | extended | unreserved | lost
pg_size_pretty(safe_wal_size) AS headroom_before_cap,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))
AS retained_wal
FROM pg_replication_slots
ORDER BY safe_wal_size ASC NULLS LAST;
-- Alerting query: slots past the cap or close to it (threshold is an example).
SELECT slot_name, wal_status, pg_size_pretty(safe_wal_size) AS headroom
FROM pg_replication_slots
WHERE wal_status IN ('unreserved', 'lost')
OR safe_wal_size < 50 * 1024^3; -- under 50 GB of headroom left
Setting max_slot_wal_keep_size, reloading and verifying it, then using wal_status and safe_wal_size in pg_replication_slots to spot slots nearing or past the cap.
What are the alternatives to capping WAL retention?
A cap is one way to bound risk, not the only one. Each option moves the failure to a different place, and the right choice depends on which outage you would rather have.
Unlimited retention guarantees the consumer never loses its position. The cost is shared risk: one stalled subscriber or CDC connector can fill the primary's disk and stop writes for every application on that cluster.
A retention cap converts that cluster-wide outage into a single-consumer outage. Once the slot is invalidated, the consumer needs a full resync. For large tables that means hours of copying, and any changes made in the meantime can leave gaps in downstream systems.
A dedicated or larger WAL volume buys time and isolates WAL from table data. It also hides abandoned slots for longer, so you pay for storage that only delays the same incident.
Automatically dropping inactive slots frees disk quickly. It cannot tell an abandoned slot from a consumer that is down for planned maintenance, so it can destroy a position you meant to keep.
Human-in-the-loop runbooks let someone check context before dropping anything. They depend on alerts firing early enough for a person to respond.
| Option | Protection | Failure mode | Operational cost | Engineering effort |
|---|---|---|---|---|
| Unlimited retention | Consumer position | Primary disk full, all apps down | Low until it fails | None |
| Retention cap | Primary availability | Slot lost, resync, data gaps | Resync time and load | Low |
| Larger WAL volume | Time margin | Delayed disk full | Storage spend | Low |
| Auto-drop inactive slots | Primary availability | Position lost during maintenance | Low | Medium |
| Runbook | Context-aware decisions | Slow response | On-call time | Medium |
How does the protection work end to end?
The controls in this article form layers, and each layer acts at a different point as a consumer falls behind.
- The slot pins WAL. Each slot records a
restart_lsn. At every checkpoint, the server works out the oldest segment it still needs, taking into account the checkpoint's redo point,wal_keep_size, and the oldestrestart_lsnacross all slots. Segments older than that point are removed or recycled. Everything newer stays inpg_wal. - Monitoring measures the gap. The per-slot query computes
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn), which is the WAL that slot is holding. Your alerting system samples it on a schedule, compares it with warn and page thresholds, and checks how long each slot has been inactive. Because the alert carriesslot_name, it points directly at the consumer that stopped. - A person investigates. On-call follows the runbook. They check whether the consumer is advancing, estimate whether it will catch up, and contact the slot's owner.
- The cap acts as a backstop. If nobody resolves the problem and the slot's retained WAL exceeds
max_slot_wal_keep_size, the slot'swal_statusmoves tounreserved. At the next checkpoint, the server stops honoring that slot's claim on the excess WAL, removes those segments, and the slot becomeslost. The primary keeps accepting writes, and the cost moves to the one consumer, which must be resynchronized. - Lifecycle hygiene prevents recurrence. Tying slots to deployments and recording owners removes orphaned slots before they can start the sequence above.
The order matters. Alerts should fire well before the cap, and the cap should be set below the point where the WAL volume fills. If those thresholds are reversed, the backstop acts before anyone has a chance to intervene, or never acts at all.
Runbook: responding to a filling disk and keeping slots under control
Emergency procedure
- Identify the slot. Query
pg_replication_slotsand rank slots bypg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn). The largest value is usually the one holding WAL back. - Check the consumer. If
activeis false, nothing is reading the slot. If it is true, compareconfirmed_flush_lsnacross two samples a minute apart. A connected consumer that is not advancing is no better than a disconnected one. - Estimate catch-up. Compare the consumer's advance rate with the WAL generation rate and the remaining free space. If the backlog is shrinking and will clear before the disk fills, fix the consumer and leave the slot alone.
- Drop it if it cannot catch up. If the slot is active, stop the consumer first, or terminate its
active_pid, because PostgreSQL will not drop an active slot. Then callpg_drop_replication_slot. Dropping the slot permanently discards the consumer's position, so it will need a full resync. Tell the consumer's owner before you drop it, not afterward.
Lifecycle hygiene
- Tie each slot to a deployment. Create the slot in the same change that ships the consumer, and remove it in the change that decommissions the consumer. Most orphaned slots come from consumers that were retired while their slot stayed behind.
- Record an owner for every slot, either in its name or in a registry. During an incident, this tells you who to ask before dropping it.
- Audit on a schedule. Flag any slot without a known owner. An occasional audit costs far less than a recovery from a full disk.
Failover behavior and slot invalidation semantics vary across PostgreSQL versions and managed services. Check your platform's documentation before relying on either.
How can this save engineering time?
Most of the time lost in a slot-driven disk incident goes to diagnosis and recovery, not to the fix itself. The controls above reduce both.
- Shorter diagnosis. A generic disk alert starts an investigation: is it WAL, table bloat, logs or temporary files? A per-slot alert names the slot and therefore the consumer, so on-call starts at the cause.
- Fewer cluster-wide outages. Without a cap, a stalled consumer can stop writes on the primary. That pulls in every team whose application depends on the cluster and may require emergency volume resizing or a restart after space is freed. With a cap, the worst case is a resync of one consumer, handled by its owner.
- Faster decisions on dropping slots. When every slot has a recorded owner, on-call does not need to find out who created an unexplained slot before deciding whether it is safe to drop.
- Less recurring work. Creating and removing slots in the same changes that ship and retire consumers prevents orphaned slots, which are the cause that is easiest to eliminate completely.
- Low setup cost. The building blocks are a reloadable parameter and queries against a system view your monitoring agent can already reach. Generating alert rules from the slot list avoids hand-maintaining them as slots change.
The trade is a small, ongoing cost (alert upkeep, scheduled audits, an ownership record) against an occasional but disruptive incident that involves several teams. Where that balance lands depends on how many slots you run and how many teams own the consumers. The more slots and owners a cluster has, the more the per-slot approach pays off.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to replication slot wal growth, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Adding or removing a column goes through one review with the inheriting services listed, instead of ad-hoc messages to each team.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Services that depend on a column get the chance to object before it disappears, avoiding emergency fixes.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.