Background Jobs and Queues 
The Half of Your System Nobody Designs 

Almost every part of a SaaS platform gets designed. The API has a spec, the data model has a review, the frontend has a figma file. The background job system has none of that — it starts as one function called after a request, gets a queue library bolted on in month four, and accumulates thirty job types over two years without anyone ever writing down what happens when one of them fails. Then a customer reports that an export never arrived, or an invoice was generated twice, or a welcome email went out nine days late, and the team discovers that the least-designed part of the system is now responsible for most of the things customers actually notice.

Why async work becomes the fragile part

The reason is structural, not cultural. A synchronous request has a client waiting on the other end, so failure is immediate, visible, and attributed — someone sees a 500 and someone gets paged. A background job fails into silence. Nobody is holding the connection, the error lands in a log that nobody reads unless they are already looking, and the business consequence surfaces days later through a support ticket that does not mention the job at all. On top of that, jobs run in a fundamentally harsher environment than request handlers: they get retried, they run concurrently with themselves, they execute against data that has changed since they were enqueued, they get killed mid-flight when a worker is redeployed, and they run at three in the morning when the third-party API they depend on is in a maintenance window. Code that is perfectly correct as a request handler is frequently incorrect as a job, and the difference only shows up under conditions that are hard to reproduce locally.

The properties that actually matter

• Idempotency. Every job will run more than once — after a retry, after a worker crash between doing the work and acknowledging it, after an operator replays a dead-letter queue. A job that charges a card, sends an email, or increments a counter must carry a key that makes the second execution a no-op.
• At-least-once delivery, assumed explicitly. Exactly-once is not available to you, and designing as though it were is the single most common source of duplicate side effects.
• Bounded retries with exponential backoff and jitter. Unbounded retries turn a downstream outage into a self-inflicted denial of service; simultaneous retries without jitter turn it into a thundering herd.
• A dead-letter destination with a replay path. Jobs that exhaust their retries must land somewhere a human can inspect, fix, and re-run — not vanish.
• Visibility timeouts longer than the worst-case job duration. A timeout shorter than the job means two workers run it concurrently, which is how a slow job becomes a duplicate-charge incident.
• Per-entity ordering where the domain requires it, and no global ordering promise anywhere else. Global ordering costs throughput and is almost never what the business actually needs.
• Isolation between workloads, so that a slow bulk-import job cannot starve password-reset emails.

One queue is one shared failure

The default arrangement — a single default queue with a pool of generic workers — works until the day a customer triggers a fifty-thousand-row import. Those jobs are slow, they occupy every worker, and everything else in the system now sits behind them: the password reset that a user is waiting on, the webhook delivery a customer's integration depends on, the nightly billing run. This is a resource-isolation failure dressed up as a performance problem, and adding workers only postpones it. The fix is to separate queues by the characteristic that actually differs, which is usually latency expectation rather than business domain. Interactive work that a human is waiting for belongs on its own queue with its own workers and a short target. Bulk and batch work belongs somewhere it can be slow without consequence. Scheduled work belongs somewhere it cannot collide with a traffic spike. A useful extra dimension in multi-tenant systems is fairness: without it, one large customer's bulk operation is indistinguishable from a system-wide outage for everyone else, and the remedy — round-robin across tenants, or a per-tenant concurrency cap — is far simpler to add before you need it than during the incident.

The enqueue-inside-transaction bug

There is one bug in this space that nearly every team ships at least once, and it is worth naming precisely because it is invisible in testing. You write a record to the database and enqueue a job to process it, in that order, inside a request. Sometimes the transaction has not committed when the worker picks the job up, and the worker cannot find the row — a race that appears only under load and looks like phantom data corruption. Or you enqueue first and the transaction then rolls back, so a job now exists for a record that does not. The standard remedy is the transactional outbox: write the intent to an outbox table in the same transaction as the business change, and have a separate relay publish from that table to the queue. The database transaction becomes the single point of truth about whether the work should happen, and the relay guarantees it eventually does. If that is more machinery than the situation warrants, the lightweight version is to make the queue itself part of the transaction — which is exactly what a database-backed queue gives you for free, and one of the strongest arguments for using one for as long as you can.

Choosing the technology

The honest default for most growth-stage teams is a queue backed by the database you already run. Postgres with SKIP LOCKED handles thousands of jobs per second, keeps enqueue transactional with your writes, gives you SQL to inspect and fix the queue during an incident, and adds no new operational surface. That is a genuinely large set of advantages, and teams usually leave it too early. A dedicated broker earns its place at higher volume or when the durability model matters: a managed queue such as SQS removes the operational burden entirely and is hard to beat for straightforward work distribution, at the cost of losing transactional enqueue and any ability to query the queue. Redis-backed queues are fast and pleasant to work with but require care about persistence semantics, because the default configuration can lose jobs on failover and most teams discover this during their first failover. A log such as Kafka is a different tool solving a different problem — it is for event streams with multiple independent consumers and replay, not for task distribution, and adopting it as a job queue means inheriting partition management and consumer-group semantics you did not need. The right question is not which is best but which failure you would rather operate: losing a job, running one twice, or running an extra database you already know how to run.

Operating it: the things worth monitoring

• Queue depth per queue, with an alert threshold — the single most predictive signal that something is wrong, and usually the earliest.
• Oldest-message age, which catches the case where depth looks fine because throughput matches arrival while individual jobs starve.
• Job duration percentiles per job type, so a job that has quietly gone from two seconds to two minutes gets noticed before it saturates the pool.
• Failure rate and dead-letter volume per job type, alerted separately — an aggregate failure rate hides a single job type failing every time.
• Worker utilisation and concurrency headroom, so scaling decisions are based on saturation rather than on queue depth alone.
• Scheduled-job execution confirmation. The nightly job that silently stopped running two weeks ago is a distressingly common discovery, and the only defence is alerting on absence, not on failure.

Where this connects to the rest of the system

Background jobs sit underneath several features people think of as separate. Reliable webhook delivery is a job queue with a specific retry policy and a customer-visible contract. Event-driven architecture is a superset of the same concerns with an additional publish-subscribe dimension. Usage aggregation for consumption billing is a scheduled job whose correctness ends up on an invoice. Search indexing, notification fan-out, report generation, and data exports are all the same machinery wearing different names. That is precisely why it is worth designing once, properly, rather than five times badly: the failure modes are identical, and a team that has solved idempotency, retries, isolation, and observability once can add the sixth async feature in an afternoon instead of a fortnight.

How Infiniti Tech Partners approaches this

Async reliability work is usually prompted by a specific embarrassment — a duplicate charge, a batch of emails that never went out, an import that took the site down — and the engagement is short and unusually concrete. We start by inventorying every job type against three questions: is it idempotent, what happens when it fails, and who finds out. That inventory alone typically reveals a handful of jobs with real side effects and no protection against re-execution, and one or two that have not run successfully in weeks. From there the work is mostly mechanical: idempotency keys where money or messages are involved, an outbox where enqueue and commit can disagree, queue separation so interactive work stops queueing behind bulk work, a dead-letter path with a replay tool an on-call engineer can actually use, and the six metrics above wired into your existing monitoring. We tend to leave the technology choice alone unless it is the actual problem, because most teams do not have a queue-technology problem — they have thirty undesigned jobs on a queue that happens to work. If any of that sounds like your system, it is a well-scoped piece of work with a visible result at the end of it.