Event-Driven Architecture: Queues, Streams, and Serverless Functions

Why decouple in time at all

Synchronous request-response is simple and correct for a great deal of software: a caller asks, waits, and gets an answer. It also couples the caller's fate to the callee's. If the downstream service is slow, the caller is slow; if it is down, the caller fails; and scaling one forces you to scale the other. Event-driven architecture breaks that coupling by inverting the flow. Instead of a service calling another and waiting, it emits an event — "an order was placed," "a file was uploaded" — and other services react on their own schedule.

The payoff is temporal decoupling: producers and consumers no longer have to be up, fast, or scaled together. A traffic spike is absorbed by a buffer instead of cascading into failure; a slow consumer falls behind and catches up rather than breaking the producer; new consumers subscribe to an existing event without the producer knowing they exist. The cost is that you trade the easy, linear reasoning of a synchronous call for a distributed system with new failure modes — and the first step to getting it right is choosing the correct backbone.

Queues, streams, and pub/sub are not interchangeable

These three are casually treated as "messaging" and they behave very differently. Choosing wrong is the most common architectural mistake in event systems.

• Queues (point-to-point) hold work for consumers to process. A message is delivered to one consumer, which processes and deletes it — the message is gone once handled. Queues are for distributing work: many workers pull from one queue and share the load, and the queue smooths bursts into steady processing.
• Publish/subscribe fans one event out to many independent subscribers, each getting its own copy. Pub/sub is for broadcasting a fact — one "order placed" event notifies billing, inventory, and analytics simultaneously, none aware of the others.
• Streams are an append-only, ordered, retained log of events. Unlike a queue, reading does not consume — many consumers read the same log independently at their own position, and events persist for a retention window so consumers can replay history or a new consumer can process from the beginning. Streams are for ordered, replayable event history and high-throughput ingestion.

The quick test: need to distribute work to workers, use a queue; need to notify several services of something that happened, use pub/sub; need ordered, replayable, high-volume event history, use a stream. Managed services exist for each across the major clouds, and event-router services can sit in front to route by content.

Serverless functions as the reaction layer

Event-driven and serverless are natural partners. A function that runs only in response to an event, scales with event volume, and costs nothing at rest is an ideal consumer — no server sitting idle waiting for work. The pattern is pervasive: an object lands in storage and triggers a function to process it; a message arrives on a queue and a function drains it; a stream feeds a function that reacts per record.

Two things to keep in mind when functions are the reaction layer:

• Concurrency and downstream limits. Functions scale out fast, which is a feature until they overwhelm a database or an API that cannot scale as quickly. Bound concurrency so the reaction layer does not become a self-inflicted denial of service on your own dependencies.
• Cold starts and duration. A function that has been idle pays a startup cost, and every function has a maximum runtime. Long or latency-sensitive processing may belong in a container consumer rather than a function.

Delivery guarantees and the idempotency you cannot skip

This is where event systems are won or lost. Distributed messaging almost universally gives you at-least-once delivery: the system guarantees a message is delivered, but under retries, redeliveries, and network faults it may be delivered more than once. "Exactly-once" is frequently marketed and, end-to-end across independent systems, largely a myth — what exists is at-least-once delivery plus consumers that handle duplicates.

That makes idempotency non-negotiable. A consumer must produce the same result whether it processes a message once or five times:

• Deduplicate on a business key. Track processed message or event IDs and skip ones already handled, so a redelivery is a no-op.
• Make operations naturally idempotent where you can — "set status to shipped" is safe to repeat; "increment balance" is not.
• Do not assume ordering unless the transport guarantees it. Queues and pub/sub generally do not preserve global order; streams preserve order only within a partition. If order matters, design around a partition key or sequence numbers explicitly rather than hoping.

A consumer that is not idempotent will eventually double-charge a customer or duplicate a record. Build it in from the first consumer, not after the first incident.

Failure handling is the architecture

In synchronous code, a failure returns an error to a caller who decides what to do. In an event system, the producer is long gone — so what happens to a message that cannot be processed is the design.

• Retries with backoff. Transient failures — a brief downstream outage — should be retried with increasing delay, not hammered instantly.
• Dead-letter queues. A message that keeps failing must not block the queue behind it or retry forever. After a retry limit, route it to a dead-letter queue for inspection, so one poison message does not stall the pipeline.
• Poison-message isolation. Malformed or un-processable events need to be quarantined and surfaced, not retried into an infinite loop that burns cost and hides the problem.
• Backpressure. When consumers cannot keep up, the buffer grows. Monitor queue depth and stream lag as first-class health signals — a rising backlog is the earliest sign of trouble in an event system, well before anything visibly breaks.
• Observability across the async boundary. A request that fans out through several events is hard to trace. Propagate correlation IDs through events so one business transaction can be followed across every hop.

When not to go event-driven

Event-driven architecture is powerful and genuinely harder to reason about, debug, and observe than synchronous calls. It is the wrong default when:

• You need an immediate answer to return to a user — a synchronous call is simpler and correct.
• The workflow is a simple, linear sequence of two or three steps with no fan-out, buffering, or independent scaling need.
• Your team has no appetite for the operational maturity — idempotency, dead-letter handling, distributed tracing — that async systems require to stay debuggable.

Reach for events where you genuinely need decoupling, buffering, fan-out, or independent scaling — not because the pattern is fashionable.

Where to start

Pick one workflow where synchronous coupling actually hurts — a spike that overwhelms a downstream service, or a step that should not block the user — and introduce a single queue or stream there. Choose the transport by what you need (work distribution, fan-out, or replayable history), make the first consumer idempotent from the start, and wire a dead-letter queue and backlog monitoring before you go live. Prove the pattern on one boundary, learn the operational habits it demands, then extend it where it earns its keep.

Event-driven systems reward getting the fundamentals — delivery semantics, idempotency, failure handling — right early, and they run on well-designed cloud infrastructure built for them. If you want help designing an event architecture that stays debuggable at scale, talk to our team.