What is the max_connections budget and why does it break on Kubernetes?

max_connections is a single server-wide ceiling. Every client that connects, whether it's an API pod, a cron job, a migration runner or a developer's psql session, takes one slot from the same budget. A few slots are held back for superusers, and changing the value requires a server restart. When the budget is gone, new connections fail with FATAL: sorry, too many clients already.

The limit exists because each connection is a separate backend process. Each process has its own memory for catalog caches and session state. Each also gets work_mem for every sort or hash step a query runs, so one complex query can use several multiples of it. Idle connections cost memory too, and every active one competes for CPU and locks.

Kubernetes makes this budget easy to exceed without noticing. The pool size set in application config applies to each pod, so the real load on the database is that number multiplied by the replica count. The Horizontal Pod Autoscaler raises the replica count at peak load, which is exactly when the database is busiest. During a rolling update, surge pods run alongside the old pods, and terminating pods may still hold their connections. The highest connection count therefore often occurs during a deploy, not during normal traffic.

After a "too many clients" error, the obvious fix is to raise max_connections. That does not add throughput, because the database still has the same cores, memory and disk. It changes how the system fails. Instead of rejecting connections quickly and visibly, it slows down through CPU oversubscription and lock contention, or the kernel's OOM killer ends a backend process. When a backend dies that way, Postgres resets all of its sessions.

This checklist is for backend and platform engineers whose services share one database and therefore draw from one connection budget.

How do you calculate total possible connections across a fleet?

That budget has to cover the worst moment, not the average one. For each workload, the ceiling is (HPA maxReplicas + rolling-update maxSurge) × pool max per pod × processes per pod. Processes matter because a pre-fork server or a multi-worker runtime opens one pool per process, not per pod. Sum that across workloads, then add the clients that bypass your pools: migration jobs, cron jobs, admin psql sessions, monitoring agents, and backup or replication tooling that connects as an ordinary client.

Illustrative estimate. Every input below is assumed. Assume max_connections = 200 and about 24 slots for overhead (migrations, cron, admin, monitoring, backup).

Service Steady replicas maxReplicas + surge Pool × processes Steady conns Worst-case conns
api 4 10 + 3 10 × 1 40 130
worker 3 6 + 2 5 × 2 30 80
reports 2 4 + 1 8 × 1 16 40
Total incl. overhead 110 274

At steady state the fleet uses 110 of 200 slots. If a deploy lands while the HPA is scaling out, surge pods open their pools before the old pods close theirs, and demand reaches 274. New clients then receive "too many clients".

Sizing to the worst-case ceiling makes exhaustion impossible by construction, but it pushes you toward either a high max_connections, which costs per-backend memory, or very small pools. Sizing to observed peaks uses resources more efficiently, but it is a bet that deploys and scale-outs never coincide. If you size to observed peaks, enforce that bet with caps, such as a lower maxReplicas or a smaller maxSurge, rather than leaving it to hope.

How should you size max_connections from what the database can serve?

…than hoping the database absorbs the spike.

Size from the server side. The useful ceiling is the number of queries the database can execute at once without them queuing on CPU or storage. A starting point is a small multiple of CPU cores, adjusted upward if queries spend much of their time waiting on I/O and your storage handles concurrent requests well. Connections beyond that number add contention and context switching without adding throughput, and each idle backend still holds memory.

Memory is the larger risk. work_mem is a per-operation limit, not per-connection. One query with several sorts and hash joins can use it several times over, hash operations can use a multiple of it, and parallel workers each get their own allowance. If every allowed connection ran such a query at the same moment, total memory use could exceed what the server has, so check that worst case against RAM.

Reserve headroom so you are not locked out. superuser_reserved_connections keeps slots open for administrators, and replication and backup connections also need slots; check how your version counts WAL senders against max_connections. Per-role and per-database caps (ALTER ROLE ... CONNECTION LIMIT, and the equivalent on ALTER DATABASE) stop one service from taking every slot. If you set those caps from steady-state usage, a normal rollout surge will hit them.

Changing max_connections requires a server restart, and hot standbys must be set at least as high as the primary. Managed services often derive the default from instance size, so resizing the instance can change it. Plan the change as maintenance. Do not ship it as a hotfix.

-- 1. Current limits and headroom settings (pending_restart = true means a change is waiting for a restart)
SELECT name, setting, unit, context, pending_restart
FROM pg_settings
WHERE name IN ('max_connections', 'superuser_reserved_connections',
               'max_wal_senders', 'work_mem', 'max_parallel_workers_per_gather');

-- 2. Peak usage by role and database, sampled during a rollout, not at steady state
SELECT usename, datname, state, count(*) AS backends
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY usename, datname, state
ORDER BY backends DESC;

-- 3. Existing per-role and per-database caps (-1 means unlimited)
SELECT rolname, rolconnlimit FROM pg_roles WHERE rolconnlimit <> -1;
SELECT datname, datconnlimit FROM pg_database WHERE datconnlimit <> -1;

-- 4. Cap each service so one cannot take every slot.
--    Leave room above the observed rollout peak, not just the steady-state count.
ALTER ROLE app_orders  CONNECTION LIMIT 60;
ALTER ROLE app_billing CONNECTION LIMIT 40;
ALTER DATABASE orders  CONNECTION LIMIT 120;

-- 5. Size the server-wide ceiling from cores and storage, then check memory.
--    Illustrative estimate with assumed inputs: 200 connections x 3 sort/hash nodes x 16MB work_mem
--    is about 9.6GB before hash multipliers and parallel workers. Compare that with RAM.
--    Count superuser_reserved_connections inside this number. Check whether your version
--    counts WAL senders against max_connections.
ALTER SYSTEM SET max_connections = 200;

-- Nothing changes until a restart. Raise hot standbys first, to a value at least as high
-- as the primary's, then restart the primary in a maintenance window.
SELECT name, setting, pending_restart FROM pg_settings WHERE name = 'max_connections';

SQL for checking current connection settings and per-role peak usage, adding role and database connection caps, and staging a max_connections change that only takes effect after a planned restart.

How do you size connection pools per pod?

Most request time is spent outside the database: parsing input, calling other services, serializing responses. A connection is held only for the queries and any open transaction. A pod handling dozens of concurrent requests therefore rarely needs more than a handful of connections at once. Set the pool maximum explicitly. Library defaults vary, and you should not let a default decide your share of the budget.

Keep the minimum idle count low, at zero or one or two. If every replica pre-opens ten idle connections, scaling from 5 to 30 pods consumes 300 slots that do nothing. A low floor lets the pool grow under load and shrink back afterward.

Set a short acquire timeout so a request that cannot get a connection fails quickly instead of hanging until the client gives up. Return a clear error, such as a 503, so load balancers and callers can back off. This is the trade-off: waiting now happens in the app's queue, not as database errors, and it is easy to miss. Export pool metrics, including wait time, pending acquires and timeouts, or the saturation stays hidden.

A generic per-pod configuration:

  • Max size: 5
  • Min idle: 1
  • Acquire timeout: 2 seconds
  • Idle timeout: 5 minutes

Cap HPA maxReplicas so that maxReplicas × pool max fits the budget from the previous step. Once the database is the bottleneck, adding pods does not add throughput. It only adds contention.

# Per-pod pool settings (Spring Boot + HikariCP), shipped as a ConfigMap.
apiVersion: v1
kind: ConfigMap
metadata:
  name: orders-api-config
data:
  application.yaml: |
    spring:
      datasource:
        hikari:
          maximum-pool-size: 5     # explicit cap: this pod's share of max_connections
          minimum-idle: 1          # low floor so idle replicas don't hoard slots
          connection-timeout: 2000 # ms; fail fast instead of hanging (map the timeout to HTTP 503)
          idle-timeout: 300000     # ms; release extra connections after 5 minutes idle
---
# Replica cap sized to the connection budget:
# maxReplicas (20) x maximum-pool-size (5) = 100 connections for this service.
# Beyond this, more pods add contention on the database, not throughput.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

A HikariCP pool configured with an explicit max of 5, one minimum idle connection, a 2-second acquire timeout and a 5-minute idle timeout, paired with an HPA whose maxReplicas keeps the total at 20 × 5 = 100 connections.

When should you add PgBouncer, and when should you avoid it?

When the sum of maxReplicas × pool max across every service still exceeds what the database can serve, a server-side pooler such as PgBouncer is the main alternative to raising max_connections. It multiplexes many client connections onto a small set of server connections. If per-pod pools already fit the budget, skip it. It adds a component to operate and does nothing for throughput.

PgBouncer's pool mode decides how much multiplexing you get:

Mode Server connection held for Multiplexing Compatibility
Session Life of the client connection Low Full
Transaction One transaction High Partial

Transaction mode gives the real savings, but consecutive transactions from one client can run on different server connections. Anything that relies on session state breaks:

  • Prepared statements: a statement prepared on one backend may not exist on the next. Check that your driver and pooler both support them.
  • Session-level SET: the setting leaks to other clients or disappears. Use SET LOCAL inside the transaction instead.
  • Advisory locks: session-scoped locks stay attached to a backend your client no longer owns.
  • LISTEN/NOTIFY: notifications need a stable session.
Sidecar:                         Central Deployment:
[pod: app -> pgbouncer] --\      [pod: app] --\
[pod: app -> pgbouncer] ---> PG  [pod: app] ---> [pgbouncer x N] --> PG
[pod: app -> pgbouncer] --/      [pod: app] --/

A sidecar removes the extra network hop, but each pod gets its own server-side pool. Server connections scale with pod count again, so you are back to the original problem. A central Deployment caps server connections at replicas × pool size. The cost is an extra hop and a new single point of failure. It needs multiple replicas, readiness and liveness probes, and its own capacity planning.

SIDECAR: one pgbouncer per pod
+-------------------------+
| pod 1: app -> pgbouncer |---+
+-------------------------+   |
+-------------------------+   |                      +----------+
| pod 2: app -> pgbouncer |---+--------------------->| Postgres |
+-------------------------+   |                      +----------+
+-------------------------+   |
| pod N: app -> pgbouncer |---+
+-------------------------+
  server conns = N pods x pool size (grows again as HPA adds pods)

CENTRAL DEPLOYMENT: shared pgbouncer tier
+------------+
| pod 1: app |---+
+------------+   |                +-----------------+
+------------+   |   +---------+  | pgbouncer x R   |           +----------+
| pod 2: app |---+-->| Service |->| probes, PDB     |---------->| Postgres |
+------------+   |   +---------+  | own capacity    |           +----------+
+------------+   |                +-----------------+
| pod N: app |---+
+------------+
  extra hop: app -> Service -> pgbouncer -> Postgres
  server conns = R replicas x pool size (independent of app pod count)

A sidecar pooler avoids the extra hop, but its server connections still scale with pod count; a central pgbouncer Deployment caps server connections at replicas × pool size but adds a hop and a tier that needs its own replicas, probes and capacity planning.

How do Kubernetes pod lifecycles leak connections?

Pod churn creates connections your fleet math never counted.

  • Budget for rolling-update overlap. With maxSurge above zero, old and new pods run together, and each holds a full pool. Size against peak replicas during a rollout, not the steady-state replica count.
  • Drain the pool on SIGTERM. Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds, then sends SIGKILL. Handle SIGTERM by failing readiness, finishing in-flight requests, and closing the pool so Postgres sees clean disconnects. A process killed mid-transaction, or a node that disappears, leaves server backends alive until something notices the dead socket.
  • Don't open a connection per readiness probe. A probe that connects to Postgres every few seconds on every pod adds constant connection churn. If the database stalls, every pod also goes unready at once. Check pool health in-process instead.
  • Let the server reclaim what clients abandon.
    • idle_in_transaction_session_timeout ends sessions that hold a transaction open and do nothing. These sessions also block vacuum and hold locks.
    • Set statement_timeout per role with ALTER ROLE ... SET so each workload gets its own limit, rather than one global value.
    • tcp_keepalives_idle, tcp_keepalives_interval, and tcp_keepalives_count control how quickly Postgres detects peers that vanished without closing the connection, such as crash-looping pods or evicted nodes.
  • Isolate long-running work. Timeouts tuned for API traffic will kill schema migrations, backfills, and batch jobs partway through. Run those workloads under a separate role with longer or disabled limits, and give them their own connection budget.
-- API role: short limits so abandoned or runaway sessions get reclaimed.
ALTER ROLE app_api SET statement_timeout = '5s';
ALTER ROLE app_api SET idle_in_transaction_session_timeout = '30s';
-- Hard cap on this role's share of max_connections (counts rollout overlap too).
ALTER ROLE app_api CONNECTION LIMIT 200;

-- Migration/batch role: no statement cap, so long backfills can finish.
ALTER ROLE app_migrator SET statement_timeout = 0;
-- Still end sessions that sit idle inside an open transaction.
ALTER ROLE app_migrator SET idle_in_transaction_session_timeout = '10min';
-- Fail fast on lock waits instead of queueing behind live traffic.
ALTER ROLE app_migrator SET lock_timeout = '10s';
-- Separate, small connection budget for jobs.
ALTER ROLE app_migrator CONNECTION LIMIT 10;

-- Server-wide TCP keepalives: detect peers that vanished without closing
-- (evicted nodes, SIGKILLed pods). Here roughly 60s + 6 * 10s before the
-- kernel gives up; illustrative values, tune to your network.
ALTER SYSTEM SET tcp_keepalives_idle = 60;
ALTER SYSTEM SET tcp_keepalives_interval = 10;
ALTER SYSTEM SET tcp_keepalives_count = 6;
SELECT pg_reload_conf();

-- Role settings apply only to new sessions; recycle pools to pick them up.
-- Verify what each role will get:
SELECT r.rolname, r.rolconnlimit, s.setconfig
FROM pg_roles r
LEFT JOIN pg_db_role_setting s ON s.setrole = r.oid
WHERE r.rolname IN ('app_api', 'app_migrator');

Per-role timeouts and connection limits keep API sessions on short limits while migrations run under a separate role with longer ones, and server-wide TCP keepalives let Postgres reclaim connections from pods that disappeared without disconnecting.

Which signals show connection exhaustion before it happens?

Exhaustion builds up gradually before the first "too many clients" error appears. These checks make the buildup visible while there is still time to act.

  • Set application_name per service. Put it in each service's connection string, or set the PGAPPNAME environment variable in the pod spec. Without it, every connection looks the same and you cannot tell which deployment is using the budget.
  • Group pg_stat_activity by application_name, usename and state. Filter on backend_type so background workers are excluded. A growing idle count points to oversized pools. A growing idle in transaction count points to application bugs that hold locks. Rising active counts mean real load.
  • Alert on used connections as a ratio of max_connections. Subtract superuser_reserved_connections first, because ordinary roles can never use those slots. Warn early enough to leave headroom for a rolling deploy, when old and new pods are connected at the same time.
  • Track pool wait time and acquire timeouts on the client. The server only sees connections that were opened. When pools are capped correctly, saturation shows up as requests queuing inside the application, and the server never sees it. Rising wait time is usually the earliest signal.

Without a budget and these signals, each incident gets the same response: raise the limit, restart the database, then spend hours reconstructing which deploy caused the connection storm. With per-service attribution and ratio alerts, one dashboard query names the service at fault. Limit changes become planned capacity decisions, and you stop discovering the problem during an outage.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to postgres max_connections kubernetes, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.