Event-Driven Architecture 
Queues, Streams, and When Async Actually Helps 

Somewhere on the path from a monolith to a set of services, most teams hit the moment where synchronous request/response stops being enough. A user action needs to trigger five downstream effects, one of them is slow, another belongs to a team you don't control, and suddenly a single HTTP request is only as reliable and as fast as the flakiest thing it calls. Event-driven architecture is the answer to that specific problem: instead of calling everything inline and waiting, you emit an event saying what happened and let interested consumers react on their own time. Done well, it decouples teams, absorbs load spikes, and makes systems more resilient. Done reflexively, it scatters your logic across a dozen consumers, replaces easy-to-debug stack traces with distributed guesswork, and trades one hard problem for several. Knowing which outcome you'll get comes down to understanding the tools and being honest about when you actually need them.

A queue is not a log — pick the right one

The most consequential and most muddled choice is between a work queue and an event stream, because they solve different problems and people reach for the wrong one constantly. A queue (SQS, RabbitMQ) distributes tasks: a message is delivered to one worker, processed, and deleted — it's ideal for offloading work like sending email, generating a PDF, or processing an upload, where you want it done once and don't care about history. An event stream or log (Kafka, Kinesis) is an append-only record that many independent consumers read at their own pace and can replay from the past — it fits when several systems need to react to the same facts, or you want an auditable history of what happened. The tell is fan-out and replay: if one consumer does one job and then it's gone, you want a queue; if multiple consumers need the same events, or you'll want to reprocess history when you add a new consumer next year, you want a log. Choosing a stream because it's fashionable when a simple queue would do is one of the most common and expensive overcomplications in this space.

The dual-write problem, and the outbox that solves it

The subtlest bug in event-driven systems is also the most common: you update your database and then publish an event, and sometimes the database commit succeeds but the publish fails (or vice versa), leaving your data and your events permanently out of sync. This dual-write problem can't be waved away with a try/catch because the process can die between the two steps. The standard fix is the transactional outbox: within the same database transaction that changes your data, you write the event into an `outbox` table, so either both happen or neither does. A separate process then reads that table and publishes the events reliably, retrying until they land. It's a small amount of extra machinery, but it's the difference between an event system you can trust and one that silently drifts — and 'silently drifts' is the worst property a distributed system can have.

Assume redelivery: idempotency and dead-letter queues

Every serious message system delivers at-least-once, which is a polite way of saying the same message will occasionally arrive twice — after a retry, a rebalance, or a consumer crash mid-processing. If processing a message twice charges a card twice or sends two emails, you have a production incident waiting to happen, so consumers must be idempotent: give each message a stable ID and make handling it a second time a safe no-op, typically by recording processed IDs or using upserts. The other non-negotiable is a dead-letter queue: when a message fails repeatedly, it must move aside after a bounded number of retries instead of blocking the queue forever or getting silently dropped, so a human can inspect and replay it. Skipping the DLQ is how one malformed message takes down an entire pipeline, and skipping idempotency is how a retry storm turns a blip into a data-integrity mess. These aren't optional polish; they're the price of admission for async.

The guarantees people assume they have and don't

Async buys resilience but takes back guarantees that synchronous code gave you for free, and teams get burned by assuming those guarantees still hold. Ordering is the classic one: across partitions and parallel consumers, messages are not globally ordered, so if your logic depends on event B never being processed before event A, you have to design for it explicitly — usually by keying related events to the same partition — rather than hoping. 'Exactly-once delivery' is largely a myth in the general case; what you actually build is at-least-once delivery plus idempotent consumers, which achieves exactly-once effects without the impossible promise. And the whole system is eventually consistent by construction: right after a user acts, the read side may not reflect it yet, which is fine for many flows and quietly wrong for others, so it's a UX and product decision, not just a technical one. Naming these trade-offs up front is what keeps event-driven systems debuggable instead of mysterious.

When not to reach for it

The most valuable discipline is restraint, because event-driven architecture makes some things dramatically better and other things dramatically worse. If a workflow is genuinely synchronous — the user needs the result now and can't proceed without it — forcing it through a queue just adds latency and complexity for no benefit. If you have one service and three developers, most of the coupling that events solve doesn't exist yet, and a well-placed background job for the few slow tasks is plenty; you don't need a broker and a schema registry to send welcome emails. Reach for async when you have real decoupling needs across teams or services, genuine load spikes to absorb, long-running work to offload, or multiple systems reacting to the same facts — and when you do, invest in the observability to trace a message across consumers, because a distributed flow you can't follow end to end is a distributed flow you can't operate. Async is a tool for specific problems, not a default architecture, and treating it as the latter is how teams manufacture the complexity they were trying to avoid when they broke up the monolith.

How Infiniti Tech Partners builds event-driven systems

We start by asking whether a workflow actually needs to be asynchronous, because the cheapest event system is the one you didn't build. When decoupling, load absorption, or fan-out genuinely justify it, we pick the right primitive — a work queue for offloaded tasks, an event log when many consumers need the same facts and replay matters — rather than defaulting to whichever is trendy. Every pipeline we ship has the reliability machinery that async demands: the transactional outbox so data and events can't drift, idempotent consumers so redelivery is safe, dead-letter queues so one bad message can't sink the flow, and end-to-end tracing so you can follow a message across every consumer. The result is a system that gets the resilience and decoupling benefits of events without the debugging nightmare that makes teams regret adopting them.