Webhooks That Don't Lose Events 
Building Outbound Events Customers Trust 

Webhooks are the cheapest integration surface a SaaS product can offer and the one most consistently shipped as an afterthought. The first version is almost always the same: somewhere inside a request handler, after the database commit, a loop that POSTs a JSON payload to every URL a customer has registered. It works in the demo, it works in staging, and it works in production right up until a customer's endpoint is slow, or down, or returns a 500 for four hours during their own deployment — at which point events are silently gone, your own API latency has doubled because you're waiting on someone else's server, and a support ticket arrives that reads 'we never received the invoice.paid event.' Webhooks look like an HTTP call. They are a distributed system with delivery guarantees, and the guarantees are the product.

Why the naive implementation fails

Three separate faults compound in the version most teams ship first. The delivery happens inside the request path, so your API's response time is now coupled to the availability of every customer endpoint you send to — one unresponsive receiver with a 30-second timeout degrades a synchronous handler for everyone. The delivery is attempted exactly once, so any transient failure is a permanent data loss from the customer's point of view, and they have no way to know it happened. And the event is generated from in-memory state at the moment of the call rather than from a durable record, which means that when it fails there is nothing to retry — the event never existed anywhere you can query. The fix for all three is the same shape: writing the event down first, delivering it asynchronously from that record, and treating delivery as a state machine with its own lifecycle rather than a side effect of a request.

The guarantees you have to choose deliberately

Exactly-once delivery over a network you don't control is not available, so the honest choice is at-least-once with published semantics. That means a receiver will occasionally get the same event twice — during a retry after a response that timed out but actually succeeded, most commonly — and your documentation must tell them to deduplicate on a stable event ID that you promise never changes across attempts. Ordering is the other decision people defer and regret: global ordering across a customer's whole event stream is expensive and rarely what they need, while per-entity ordering (all events for a given subscription, in sequence) is achievable with a partitioned queue and is what integrations actually depend on. If you can't guarantee ordering, say so explicitly and include a sequence number or a version on the entity so receivers can discard stale updates. The related rule is that a webhook payload should carry enough to be useful but should never be the sole source of truth — a receiver that missed an event must be able to reconcile by calling your API, which means every event type needs a corresponding read endpoint. These are the same delivery semantics that govern internal event-driven systems; the difference is that the consumer is a stranger's codebase you cannot fix.

Security: signatures, replay, and the SSRF nobody mentions

An outbound webhook system makes authenticated-looking HTTP requests to URLs that customers supply, which is a server-side request forgery primitive handed to you by design. Somebody will register an endpoint pointing at your cloud provider's metadata service, or at an internal address on your own network, and if your delivery workers run without egress controls that request will succeed. Validate and re-resolve destination hostnames at delivery time, block private and link-local ranges, and run delivery workers in a network segment that cannot reach your internal services. On the receiving side, customers need to verify the request came from you: sign the raw request body with an HMAC using a per-endpoint secret, include a timestamp inside the signed payload so old captured requests can't be replayed, and publish the verification snippet in your docs — because otherwise a meaningful share of your customers will simply not check. Support two active secrets at once so rotation is possible without a coordinated outage, and never put anything in the payload that the receiving endpoint's owner shouldn't see; webhook URLs get pasted into automation tools and shared inboxes far more casually than API keys do.

What a production webhook system actually contains

• A durable event record written in the same transaction as the state change it describes, so an event is never lost between commit and enqueue.
• An asynchronous delivery worker with a bounded timeout, isolated from the request path and from your other background work.
• Exponential backoff with jitter across a defined retry window — typically hours, not minutes — and a documented schedule so customers know how long they have to recover.
• Automatic endpoint disabling after sustained failure, with notification to the customer, so one abandoned URL doesn't consume delivery capacity indefinitely.
• A dead-letter store and an operator-triggered replay path, because the first thing anyone asks after an incident is whether the events can be re-sent.
• Per-customer delivery logs with request, response status, and body — visible to the customer, so 'we never received it' becomes a question they can answer themselves.
• Rate limiting per destination, so a customer with a fragile endpoint isn't taken down by your retry storm.

The operational surface you're signing up for

Once webhooks exist, a category of support load exists with them, and it is not small. Customers will point at your system when their own endpoint returns a 502, so you need delivery logs they can inspect without contacting you — this single feature removes more tickets than any other. You'll need a way to send a test event on demand, because that's the first thing anyone does when configuring an integration. Noisy neighbours are real: a customer generating a hundred thousand events an hour shares your delivery workers with everyone else unless you partition capacity. And webhook payloads are an API contract in the strictest sense — receivers write brittle parsing against them, so the versioning discipline you apply to your REST endpoints applies here too, with additive changes only and a real deprecation process for anything else. Instrument it as a first-class service: delivery success rate, time from event creation to successful delivery, and queue depth per destination are the three metrics that tell you whether the system is healthy, and they belong on the same dashboards as your API. Backlog age is the alert that matters — a queue that is growing has already broken a promise, and knowing that thirty minutes before your customers do is the entire value of the instrumentation.

Build, or buy the delivery layer

There are now credible vendors that operate webhook delivery as a service — you publish events to them, they handle retries, signing, endpoint management, customer-facing logs, and the support surface around all of it. The trade is the usual one. Building it yourself is perhaps three to six weeks for a competent team to reach the feature list above, plus permanent ownership of an operational service that is boring when it works and urgent when it doesn't. Buying it removes that ownership but puts your customers' integration reliability behind a third party, adds a per-event cost that scales with your volume, and means one more processor in your vendor risk and data-flow inventory, since event payloads frequently contain customer data. Our general guidance is that the event record and the semantics — what an event means, when it fires, what it contains, how it's ordered — should always be yours, because they're part of your API contract, while the transport and retry machinery is a reasonable thing to rent, particularly early. What you should not do is defer the decision by shipping the for-loop, because migrating customers off an unreliable webhook implementation is considerably harder than building a reliable one to begin with.

How Infiniti Tech Partners builds event delivery

We treat outbound events as a product surface with an SLA rather than a background feature, and the engagements usually start the same way: a customer has complained about missing events, and nobody can prove whether they were sent. The first deliverable is almost always the durable event record and a delivery log the customer can see for themselves, because that converts an unfalsifiable support argument into a fact and buys the team room to fix the rest properly. From there we implement the delivery pipeline — asynchronous workers, backoff with jitter, per-destination isolation, dead-letter and replay, HMAC signing with rotatable secrets, and egress controls on the workers — and we write the receiver-side documentation, including the verification snippet and the deduplication guidance, because an integration contract that customers can't implement correctly isn't finished. If your webhooks are currently a loop inside a request handler and you'd like to know what it takes to make them dependable before an enterprise customer builds something important on top of them, that's a well-scoped piece of work with a clear end state.