Designing Backpressure into Event-Driven Systems

A pricing service publishes an event every time someone edits a catalogue. Usually that is a few hundred an hour. Then a bulk import runs and it becomes a few hundred a second. The consumer, which recalculates search indexes, keeps up for about ninety seconds, falls behind, and then the queue grows, memory climbs, and the pod is killed. The backlog is redelivered and the whole cycle starts again. Nothing in the code was wrong. The system simply had no way of telling the producer to slow down.

Backpressure is a conversation, not a queue

In a synchronous call, the caller waits. If the callee is slow, the caller is slow too, and that slowness propagates up the chain automatically. Events cut that wire. The producer hands a message to a broker and moves on, so a mismatch between production and consumption rate can only show up in the queue between them.

Backpressure is any mechanism that closes the loop again. Sometimes the signal travels all the way back and the producer genuinely slows down. Sometimes it cannot — you cannot ask thousands of mobile clients to publish less — so the system fails fast instead: reject, shed, degrade. Both are backpressure. Hoping is not.

The oldest and best-understood version is TCP: a receiver advertises how much data it can still accept, and when the window closes, the sender stops. Event systems deserve the same honesty.

Put a bound on every queue

An unbounded queue converts a latency problem into a memory problem, and it does so slowly enough that nobody notices until a pod restarts. Do the arithmetic for your own system. A worker that handles 200 events a second, with bursts of 5,000, is comfortable behind a 10,000-message buffer. The same worker facing a sustained 400 events a second for an hour is 720,000 messages behind, and the queue has quietly become an archive with a retention policy nobody agreed to.

Choose the overflow policy deliberately, and write it down:

• Block the producer. The strongest and simplest backpressure there is, provided the producer can afford to wait — an internal batch job usually can, a user-facing request usually cannot.
• Drop the newest. Cheap, and correct when fresh data supersedes stale data, such as a price tick or a position update.
• Drop the oldest. Similar, with different consequences. Make sure downstream consumers can tolerate the gap.
• Persist and drain. A durable log absorbs the burst to disk, but only if you have sized retention and disk for the worst case, and only if you accept that catching up takes time.
• Reject outright. The producer gets an error and decides what to do. Honest, and often the right answer.

"Grow the queue" is not on the list.

Bound in-flight work on the consumer side too. Prefetch counts, maximum poll records and thread pool sizes all decide how much work a consumer holds at once. A consumer that pulls 5,000 messages and processes them one at a time has taken 5,000 messages hostage and made the backlog worse for everyone else.

Rate limits belong on the consumer's edge

A rate limit is a promise about capacity. Put it as close to the constrained resource as possible — the consumer's input, not the producer's output — because the producer rarely knows how busy the consumer is.

Token buckets and leaky buckets smooth bursts, and a plain concurrency semaphore is often enough when the real constraint is database connections. Set the number from measurement: find the throughput that keeps your p99 within target, then take a slice off the top for headroom. A limit discovered during an incident is usually too high.

Retry behaviour matters more than the limit itself. Exponential backoff with jitter, a capped number of attempts, and ideally a shared retry budget so a thousand clients do not decide to retry in the same second. Add a circuit breaker so a dependency that is already failing gets a rest rather than a stampede.

Load shedding, done deliberately

There will be moments when the queue is full and the limit has been hit. Something has to give, and the only question is whether you chose it in advance.

Not all events are equal. A payment authorisation is not a recommendation refresh. Tag events with a priority, reserve capacity for the high-priority path, and shed cheap work first — dropping a telemetry batch costs nothing, dropping a payment does.

Shed early, too. A request rejected in two milliseconds is nearly free. The same request held open for thirty seconds and then timed out has consumed a connection, a thread and a client's patience. Say so clearly in the response: a 429 with a Retry-After hint, or a 503, so the caller knows that retrying immediately is exactly the wrong move.

Graceful degradation is the softer cousin: serve slightly stale cached data, skip an optional enrichment step, batch less aggressively. It keeps the system useful while it catches up.

Measure the queue, not the CPU

CPU tells you how hard a consumer is working. It does not tell you whether it is winning. Track these instead:

• Queue depth and message age. Age is the better signal: a depth of 10,000 is fine if you drain 10,000 a second, but a message forty minutes old is not.
• Consumer lag per partition or subscription. One hot key can stall a single partition while the rest sit idle.
• In-flight count. This is the difference between a busy consumer and a saturated one.
• Counters for every shed, drop and dead-letter event. If nothing counts them, they are not happening as far as your dashboard is concerned.
• Retry rate. It often starts climbing minutes before the queue does.

Alert on trends rather than spikes. Lag that has grown steadily for fifteen minutes is a far better signal than one noisy minute. As a rough sizing rule, the concurrency you need is arrival rate multiplied by processing time, which is a useful way to work out how many workers a given throughput demands.

Patterns that make things worse

• Unlimited retries. A retry storm is self-inflicted load arriving at the worst possible moment.
• Unbounded in-memory queues inside the process, which move the outage from the broker to the heap.
• One shared thread pool for fast and slow work, so a slow job starves the quick one.
• A poison message that blocks a partition forever because there is no dead-letter path.
• Autoscaling on CPU when the bottleneck is a downstream database. More consumers make that worse, not better.
• Batch sizes that grow with the backlog, turning a queue problem into a memory problem on the consumer.

A short checklist before the next deploy

Five questions worth answering out loud, with the team in the room:

1. What is the maximum size of every queue and buffer on this path, and what happens when it is reached?
2. Where is the rate limit, and what number is it based on?
3. Which events can we drop, and which can we never drop?
4. What does a producer see when we push back — a block, a 429, or silence?
5. Which metric tells us we are losing, and how long does it take to fire?

Write the answers down. Most backpressure incidents are less a failure of engineering than a failure of decision: nobody chose a limit, so the system chose one for them, at the worst possible moment.

Photo: ArtisticOperations / Pixabay