What is backpressure in asynchronous microservices, and why do queues only delay overload?

Backpressure is how a system responds when a consumer cannot keep up with the rate at which work arrives. There are three possible responses:

  • Propagate: signal upstream so producers slow down, block, or receive errors they can act on.
  • Absorb: hold the excess in a buffer with an explicit size limit, sized for bursts you expect to drain.
  • Shed: drop, reject, or defer work deliberately, and decide in advance what gets lost.

A message broker can look like a fourth option. In practice it is the second one, absorb, with the limit removed or set very high. A queue turns a rate mismatch into a growing backlog. Suppose producers send 1,200 messages per second and consumers sustain 1,000. The backlog then grows by 200 messages per second, roughly 17 million per day (illustrative estimate).

Nothing fails while this happens. Latency rises, and dashboards that track only errors stay green. The failure comes later and all at once. A RabbitMQ node hits its memory or disk limit, or Kafka's retention policy deletes records that a lagging consumer group never read. By then the backlog can take hours to drain, even after you fix the cause.

This guide is for backend and platform engineers who run Kafka or RabbitMQ consumers in production. You need to design for backpressure whenever producer rate can exceed sustained consumer throughput, even briefly. That covers almost any pipeline with bursty traffic or a downstream dependency that can slow down.

  1,200 msg/s in                                        1,000 msg/s out
 +-------------+      +-----------------------------+      +-------------+
 |  Producers  |----->|  Broker queue (RabbitMQ /   |----->|  Consumers  |
 |             |      |  Kafka topic)               |      |             |
 +-------------+      |  [#][#][#][#][#][#] ...     |      +-------------+
        ^             |  backlog +200 msg/s         |
        |             +-----------------------------+
        |                  |          |          |
        |    (1) PROPAGATE |          |          | (3) SHED
        +------------------+          |          v
          block, throttle, or         |   +-------------------+
          return errors upstream      |   | drop, reject, or  |
                                      |   | defer by policy   |
                                      |   +-------------------+
                                      | (2) ABSORB
                                      v
                      +-----------------------------+
                      | bounded buffer: explicit    |
                      | size limit, sized to drain  |
                      | expected bursts             |
                      +-----------------------------+

  No limit: latency rises, errors stay at zero, then failure at once:
 +----------------------+    +-------------------------+
 | RabbitMQ hits memory |    | Kafka retention deletes |
 | or disk limit        |    | records never consumed  |
 +----------------------+    +-------------------------+

When producers outpace consumers, the system can propagate the signal upstream, absorb the excess in a bounded buffer, or shed load; leaving the queue unbounded only defers failure until broker memory, disk or retention runs out.

Who should use these backpressure techniques?

Use them if you own any part of the path from an incoming request to a message consumer. In practice, that means these roles:

  • Backend engineers writing consumers. You choose the prefetch count, the poll loop structure, the ack and commit timing, and the retry behaviour. These choices decide whether a slow handler stays contained or spreads.
  • Engineers writing producers and APIs. You decide what happens when the broker or producer buffer is full: block, fail fast, or queue locally. This decides whether clients get a clear 503 or a request that hangs.
  • Platform and SRE teams running the brokers. You set RabbitMQ memory and disk limits, queue length limits and Kafka retention. You also own the alerts on lag and queue depth that warn you before data is lost or publishers are blocked.

The techniques matter most in these situations:

  • Traffic is bursty, for example batch imports, scheduled jobs or webhook storms.
  • Consumers depend on a database or external API whose latency varies.
  • One broker is shared by many services, so one team's backlog can block another team's publishers.
  • Losing or delaying messages has a business cost, so you need a deliberate policy rather than whatever the broker does by default.

They matter least for a low-volume pipeline where the consumer has large, steady headroom over peak producer rate and no fragile downstream dependency. Even there, a bounded buffer and a lag alert are cheap insurance.

How does backpressure work end to end?

Backpressure works by putting an explicit limit at every hop and making each full limit visible to the hop before it. No single setting does the job. It is a chain, and the signal travels upstream one link at a time:

  1. Consumer work queue. The consumer hands messages to a bounded in-process queue. When it fills, the consumer stops taking new messages. In Kafka the consumer calls pause() on its partitions. In RabbitMQ, unacked deliveries reach the prefetch limit.
  2. Broker delivery. In RabbitMQ the broker stops delivering to that consumer until it acks. In Kafka the consumer stops fetching. Either way, the excess stays on the broker.
  3. Broker storage. The backlog grows in the queue or partition log. A RabbitMQ length limit rejects or drops messages. A resource alarm blocks publishers. In Kafka, lag rises toward the retention window, and an alert on lag stands in for the limit Kafka does not enforce.
  4. Producer client. The producer sees blocked connections or confirm timeouts in RabbitMQ, or a full buffer.memory and a send() that throws after max.block.ms in Kafka.
  5. API edge. The service turns those producer failures into 429 or 503 with Retry-After, so the original caller slows down instead of piling up more work.

At each step, work is either accepted within a known bound or refused with a signal the previous hop can act on. Where a hop cannot pass the signal further back, for example when a third-party producer ignores 503, you shed load at that hop by policy. The sections below cover each link: how the two brokers differ, how to control Kafka consumers, how to size RabbitMQ prefetch, how to propagate to producers, and what to do when producers will not slow down.

How do Kafka and RabbitMQ handle slow consumers differently?

Kafka consumers pull. Each consumer fetches records from its assigned partitions when it is ready, so a slow consumer never pushes load back onto the broker. The broker keeps appending to the log regardless. Backpressure shows up as consumer lag: for each partition, the log-end offset minus the consumer group's committed offset. Lag is a distance, not a stall, and nothing upstream notices it growing.

That is why Kafka's failure mode is silent. Retention is set by time or size, not by whether anyone has read the data. If lag grows past the retention window, the broker deletes segments the group never consumed. The producer still gets successful acks. The consumer finds its committed offset out of range and, depending on its offset-reset policy, jumps to the earliest or latest available offset or raises an error. Either way, the unread data is gone.

RabbitMQ pushes. The broker delivers messages to consumers, and the prefetch count caps how many unacknowledged messages a consumer or channel can hold. Once that cap is reached, the rest of the backlog stays in the queue on the broker. As queues grow, the broker consumes memory and disk. When a memory or disk alarm fires, RabbitMQ's flow control blocks publishing connections. The alarm applies broadly, so producers that have nothing to do with the slow consumer can stall too.

Kafka RabbitMQ
Where the buffer lives Partition log on broker disk Queue in broker memory and disk
Overload signal Rising consumer lag Queue depth, then resource alarms
What fails first Unread data aged out by retention Publishers blocked by the broker
Who is affected The lagging consumer group, silently Publishers, often including unrelated ones

How do you measure and control consumer lag in Kafka?

Lag is measured per partition: the log-end offset minus the group's committed offset. kafka-consumer-groups.sh --bootstrap-server <host:port> --describe --group <group> prints CURRENT-OFFSET, LOG-END-OFFSET and LAG for every partition. Summing the lag hides problems, because one hot partition can be hours behind while the rest are idle.

A raw message count tells you little on its own. Divide lag by the recent consume rate to get time behind. For example, 600,000 messages at 2,000 messages per second is five minutes (illustrative estimate). Compare that figure with your retention period and your freshness SLO. Alert on the trend as well as the absolute value. Lag that is high but shrinking is a recovery. Lag that is low but rising steadily is the real incident.

On the consumer side, keep the poll loop thin and hand records to a bounded work queue. When the queue fills, pause() the assigned partitions and keep calling poll(). The consumer then stays in the group without fetching more records, and resume() restarts fetching once the queue drains. Pausing protects memory, but lag keeps growing while partitions are paused. If processing blocks the loop for longer than max.poll.interval.ms, the consumer is evicted. The rebalance moves its partitions to other consumers, and they slow down too, which can turn into a rebalance storm. Lower max.poll.records so each batch finishes well within that interval. Commit offsets only after processing completes, so a crash replays records instead of dropping them.

Parallelism has a hard ceiling. Each partition is consumed by at most one consumer in a group, so consumers beyond the partition count sit idle. Adding partitions later changes which partition each key maps to, and that breaks per-key ordering during the transition.

Properties props = new Properties();
props.put("bootstrap.servers", "kafka-1.internal:9092");
props.put("group.id", "orders-processor");
props.put("enable.auto.commit", "false");      // commit only after processing finishes
props.put("max.poll.records", "200");           // small batches stay well inside the poll interval
props.put("max.poll.interval.ms", "300000");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());

int maxPollRecords = 200, capacity = 2_000;
BlockingQueue<ConsumerRecord<String, String>> work = new ArrayBlockingQueue<>(capacity); // bounded buffer
Map<TopicPartition, OffsetAndMetadata> done = new ConcurrentHashMap<>();

// One worker preserves per-partition order and records the next offset to commit.
Thread worker = new Thread(() -> { while (true) { try {
  ConsumerRecord<String, String> r = work.take();
  process(r);
  done.put(new TopicPartition(r.topic(), r.partition()), new OffsetAndMetadata(r.offset() + 1));
} catch (InterruptedException e) { return; } } });
worker.start();

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
  consumer.subscribe(List.of("orders"));
  while (true) {
    // Thin loop: keep polling even while paused so the consumer stays in the group.
    for (ConsumerRecord<String, String> r : consumer.poll(Duration.ofMillis(500))) work.add(r);
    if (work.remainingCapacity() < maxPollRecords) {
      consumer.pause(consumer.assignment());     // next batch might not fit: stop fetching
    } else if (work.size() < capacity / 2 && !consumer.paused().isEmpty()) {
      consumer.resume(consumer.paused());        // drained below half: fetch again
    }
    // Commit only offsets the worker has finished; a crash replays, never drops.
    Map<TopicPartition, OffsetAndMetadata> batch = new HashMap<>(done);
    if (!batch.isEmpty()) {
      consumer.commitSync(batch);
      batch.forEach(done::remove);               // remove(k, v): keeps newer offsets added meanwhile
    }
  }
} // Production code also needs a ConsumerRebalanceListener to drain or drop queued work for revoked partitions.

A Java consumer loop that hands records to a bounded queue, pauses all assigned partitions before the queue can overflow, resumes once it drains below half, and commits only offsets whose processing has completed.

How do you choose a RabbitMQ prefetch count?

basic.qos caps how many delivered-but-unacknowledged messages the broker will hand to a consumer. When the cap is reached, RabbitMQ stops delivering to that consumer until it acks something. This makes the prefetch count the main consumer-side backpressure control. It also controls fair dispatch: a busy consumer stops receiving work, so new messages go to consumers that have spare capacity. Prefetch only applies with manual acks. With automatic acks, messages are pushed as fast as the network allows.

In RabbitMQ, global=false applies the limit to each consumer, and global=true applies it to the whole channel. Use per-consumer limits unless you have a reason not to. Quorum queues do not support the channel-wide form.

Illustrative estimate. Assume 5 ms of processing per message and a 20 ms network round trip.

  • Prefetch of 1: each message costs about 25 ms, because the consumer waits for its ack to reach the broker and the next message to arrive. That gives roughly 40 msg/s at about 20% utilisation.
  • Keeping the consumer busy: the consumer needs about (20 + 5) / 5 = 5 messages in flight. A prefetch of 8–10 adds headroom.
  • Prefetch of 1,000: one consumer holds about 5 seconds of queued work. At an assumed 50 KB per message, that is about 50 MB in its memory. If it crashes, all 1,000 messages are redelivered at once.

Config sketch. Call basic_qos with prefetch_count=10, consume with auto_ack=False, and call basic_ack only after processing succeeds.

Concern Low prefetch High prefetch
Throughput Limited by round-trip time Close to processing rate
Fairness Even across consumers Fast or early consumers hoard work
Memory Small per consumer Concentrated in one process
Redelivery after crash Few messages Large burst
import pika

credentials = pika.PlainCredentials("orders-consumer", "change-me")
params = pika.ConnectionParameters(host="rabbitmq.internal", credentials=credentials)

connection = pika.BlockingConnection(params)
channel = connection.channel()

channel.basic_qos(prefetch_count=10, global_qos=False)


def process(body):
    # Placeholder for real work, assumed ~5 ms per message.
    pass


def on_message(ch, method, properties, body):
    try:
        process(body)
    except Exception:
        # Reject without requeue so a poison message does not loop forever.
        # Route it to a dead-letter exchange if one is configured.
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)
        return
    # Ack only after the work succeeds. This frees a prefetch slot,
    # so the broker can send the next message.
    ch.basic_ack(delivery_tag=method.delivery_tag)


channel.basic_consume(queue="orders", on_message_callback=on_message, auto_ack=False)

try:
    channel.start_consuming()
finally:
    # Messages still unacked when the connection closes are requeued.
    # At most prefetch_count of them come from this consumer.
    connection.close()

A pika consumer that caps unacknowledged deliveries at 10 per consumer with basic_qos and acks each message manually after it is processed, so the broker holds back new deliveries while the consumer is busy.

How do you propagate backpressure from brokers back to producers and APIs?

Prefetch limits how fast work leaves the broker. The harder problem is limiting how fast work enters it, and that limit has to reach the service that accepted the original request.

RabbitMQ: blocked connections and confirm timeouts

When a node crosses its memory high watermark or its free-disk limit, RabbitMQ raises a resource alarm and stops reading from connections that publish. Alarms apply to the whole cluster, so a single runaway queue can stall every producer on the broker, including ones that never touch that queue. Clients that support the connection.blocked notification learn about this right away. Clients that don't simply see writes hang. Treat a blocked connection as a load signal and stop accepting new work. Do not queue messages locally while you wait.

Publisher confirms give you a second bound. Cap the number of unconfirmed messages per channel and set a deadline on each confirm. A timeout means the broker isn't keeping up, so report it as a failure. Do not retry it in a tight loop.

Kafka: bounded buffers and delivery deadlines

The Kafka producer stores records in a fixed buffer.memory. When that buffer is full, send() blocks for up to max.block.ms and then throws an exception. Keep max.block.ms short so the exception reaches your code quickly. Pair acks=all with a delivery.timeout.ms that matches your request deadline, so a write is either durable or reported as failed within a known time.

Surfacing it at the edge

Map these signals to HTTP responses. Return 429 when a specific caller exceeds its quota. Return 503 with Retry-After when the downstream path is saturated. For example, the producer's handler catches a full buffer or a confirm timeout and returns 503 immediately, without accepting the request. This bounds the amount of in-flight work at every hop between the client and the consumer.

import java.util.Properties;
import java.util.concurrent.ExecutionException;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.errors.TimeoutException;
import org.apache.kafka.common.serialization.StringSerializer;

public class OrderIngest {
  private final KafkaProducer<String, String> producer;

  public OrderIngest() {
    Properties p = new Properties();
    p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1.internal:9092");
    p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    p.put(ProducerConfig.ACKS_CONFIG, "all");                    // success means durable on all in-sync replicas
    p.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 16L * 1024 * 1024); // fixed bound on bytes waiting in this process
    p.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 50);                // fail fast when the buffer is full
    p.put(ProducerConfig.LINGER_MS_CONFIG, 5);
    p.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, 3000);
    p.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 5000);        // >= linger + request timeout; fits the HTTP deadline
    producer = new KafkaProducer<>(p);
  }

  /** Returns the HTTP status to send. On 503 the caller should also set Retry-After. */
  public int handle(String orderId, String json) throws InterruptedException {
    try {
      // get() waits at most about delivery.timeout.ms, so in-flight work per request stays bounded
      producer.send(new ProducerRecord<>("orders", orderId, json)).get();
      return 202;
    } catch (TimeoutException e) {
      return 503; // thrown synchronously: buffer stayed full (or metadata unavailable) past max.block.ms
    } catch (ExecutionException e) {
      // BufferExhaustedException and expired batches arrive here as TimeoutException causes
      if (e.getCause() instanceof TimeoutException) return 503;
      return 500; // non-retriable error: the problem is not overload, so do not tell the client to retry
    }
  }
}

A Kafka producer with a bounded buffer and acks=all reports a full buffer or an expired delivery to the HTTP handler as 503 and does not accept the request.

What are the alternatives when you cannot slow producers down?

Some producers will not slow down: third-party webhooks, device fleets, or clients that retry on 503. In those cases you decide what to drop and what to isolate.

Length limits. A RabbitMQ queue can be capped by message count or bytes. With drop-head overflow, the broker discards the oldest messages. With reject-publish, it refuses new ones, and publishers using confirms see the rejection. Kafka has no queue cap. Retention deletes old segments whether or not a lagging consumer has read them, so the loss is silent unless you alert on lag approaching the retention window.

TTLs and dead-lettering. Expire work that has lost its value, such as a quote past its validity window. Route expired or rejected messages to a dead-letter exchange so they stay inspectable.

Separate queues or topics. Give priority traffic its own queue or topic and its own consumers, so a bulk backlog cannot delay it.

Circuit breakers. If a consumer is slow because a dependency is slow, failing fast stops each message from waiting out a full timeout.

Limits and TTLs lose data by design. Get business sign-off on what may be dropped, and alert on drop, expiry and dead-letter rates.

Pattern Protects Costs Use when
Length limit Broker memory and disk Dropped or rejected messages Stale data has no value
TTL + DLQ Consumers from stale work Expired messages need triage Work has a validity window
Separate queues/topics Priority latency More consumers and config Traffic classes differ
Circuit breaker Consumer threads Failed or deferred messages A dependency is the bottleneck

When should you avoid autoscaling and retries as a fix for slow consumers?

Autoscaling on consumer lag or queue depth assumes the consumers themselves are the bottleneck. That holds when handlers are CPU-bound and independent. It fails when every consumer writes to the same database or calls the same rate-limited API. More consumers then mean more concurrent connections, more lock contention and more 429s. Throughput flattens or drops, lag keeps growing, and the autoscaler adds more instances in response. Check where time is actually spent before scaling. If handler latency is dominated by a downstream call, the constraint is downstream.

Kafka adds a hard ceiling. Within a consumer group, each partition is assigned to at most one consumer. Instances beyond the partition count sit idle, and each scaling event triggers a rebalance that pauses consumption.

Retries have a similar failure pattern. When a downstream slows, immediate retries multiply the load on it at the moment it can least absorb it. In RabbitMQ, requeueing a failed message puts it back for immediate redelivery, which can loop it continuously. Under at-least-once delivery, any retry can repeat a side effect that partially succeeded. Retries are only safe with exponential backoff and jitter, a retry cap that routes to a dead-letter queue, and idempotent handlers keyed on a message or business ID.

How can backpressure design save engineering time?

Bounding in-flight work with these controls takes up-front design and testing time. In exchange, failures behave predictably: work slows and messages park instead of cascading. The team also spends less time on incident response, manual lag recovery, and cleaning up duplicate charges or records.

The time savings come from several places:

  • Shorter incidents. An unbounded backlog has to be drained after the cause is fixed, and drain time depends only on spare consumer capacity. Assume a 1,000,000-message backlog and consumers with 200 msg/s of headroom above the incoming rate. Draining it takes about 5,000 seconds, or roughly 83 minutes (illustrative estimate). With bounded buffers and shedding, the backlog stays small, so recovery ends soon after the cause is fixed.
  • Faster diagnosis. A 503 from the ingest API, a paused partition or a connection.blocked event points directly at the saturated hop. Without these signals, engineers start from rising latency and work backwards through every service.
  • Less data repair. Committing offsets after processing, acking after success and using idempotent handlers mean a crash replays work safely. Nobody has to reconstruct records lost to retention, or find and reverse duplicate side effects by hand.
  • Fewer cross-team escalations. Per-consumer prefetch limits, queue length limits and separate queues for priority traffic keep one service's backlog from blocking another team's publishers through a cluster-wide resource alarm.
  • Less trial-and-error tuning. When lag is measured as time behind and compared with retention and the freshness SLO, capacity decisions follow from numbers your system already reports. That replaces repeated rounds of adding consumers and watching whether lag moves.

These savings depend on the alerts being in place: lag trend per partition, queue depth, drop and dead-letter rates, and the rate of 503 responses at the edge. Without them, bounded systems still fail predictably, but the team finds out from users instead of from monitoring.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to backpressure microservices kafka, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.