Reliable webhooks: surviving duplicates, timeouts and traffic spikes on AWS and Azure

A stable webhook does more than return 200. It defines an operational contract: quick acknowledgement, durable storage, duplicate handling, controlled retries, DLQ and observability. Learn practical AWS and Azure patterns for systems under real load.

The full article is available below. Enable JavaScript to use quizzes and simulations.

Reliable webhooks: surviving duplicates, timeouts and traffic spikes on AWS and Azure

A webhook fails because it is a boundary between distributed systems with different timeouts, delivery semantics and tolerance for disorder, not because it is a small HTTP endpoint.

The central argument

A reliable webhook takes more than a controller that quickly returns HTTP 200. It is a deliberately designed flow: verification, durable storage, deduplication, a queue, retries, dead-lettering, concurrency control and observability. Without those, it works only when the surrounding world behaves kindly. Production does not.

Many projects treat webhooks as ordinary REST endpoints: receive a request, execute all business logic and respond. This lasts until the first timeout, sender retry, traffic spike or duplicate event.

A webhook is an operational contract between systems that rarely share guarantees. One assumes at-least-once delivery, another expects exactly once. One retries aggressively, another cannot handle duplicates. One sends concurrently, another quietly assumes order. This is where real architectural problems begin.

The common mistake: assuming a successful HTTP response means business success. It does not.

Key takeaways

A webhook should acknowledge receipt quickly and perform expensive work asynchronously.
Duplicates are normal in distributed systems.
Event ordering must be explicitly designed.
Retries without idempotency and a DLQ postpone failure rather than establish reliability.
Without event IDs, retry counts and correlation IDs, diagnosis becomes guesswork. Observability is part of webhook stability.

Why webhooks fail in production

Failure usually comes from several small assumptions combining into an expensive problem:

the event will arrive only once,
it will arrive in order,
downstream will respond quickly,
the worker will finish before visibility timeout or lock expiry,
one successful run means the architecture is good enough.

AWS and Azure offer ingress limits, retrying queues, dead-lettering, idempotent functions, concurrency controls and recovery paths. Problems arise when these mechanisms exist but are not connected into one coherent operating model.

A stable webhook is a set of guarantees

A well-designed webhook follows these principles:

Verify — validate the signature, source, timestamp and message structure.
Persist — durably save the accepted event or its ledger before side effects.
Deduplicate — make the event ID an architectural key.
Enqueue — separate HTTP ingress from business work using a queue or bus.
Acknowledge fast — return 200/202 promptly rather than doing everything inside the request.
Process idempotently — make workers safe under retries, duplicates and replays.
Dead-letter explicitly — route failed cases into controlled recovery.
Observe everything — give each important step a trace, metric and correlation ID.

An AWS reference pattern

A practical inbound webhook model on AWS is:

API Gateway → Lambda ingress → SQS → Lambda / ECS / Fargate worker → storage + outbox + DLQ

This organizes responsibilities.

1. API Gateway as the entry gate

API Gateway controls incoming traffic, rejects obvious abuse, applies throttling and supports predictable behavior during bursts. If ingress is already overwhelmed, even an excellent worker cannot help.

2. Lambda ingress as a minimal layer

Ingress should avoid expensive business logic. It verifies the signature, extracts the event ID, records the idempotency ledger, enqueues the message and responds quickly.

3. SQS separates responsibilities

Choose Standard when throughput and flexibility matter more than strict ordering. Choose FIFO when ordering for one aggregate matters and you need message groups and deduplication within FIFO's guarantees.

4. The worker owns domain logic

Database updates, side effects, state-transition validation, outbox use and controlled acknowledgement belong here. A worker must not assume this is the first delivery or that events arrived in perfect order.

5. DLQ as an operational layer

A dead-letter queue records cases the system could not finish automatically. It provides material for analysis, replay, correction and architectural improvement.

An Azure reference pattern

A sensible Azure model often looks like this:

Front Door / API Management → Azure Functions HTTP trigger → Service Bus → Function worker / Container Apps / AKS → storage + DLQ + Application Insights

1. API Management or Front Door at ingress

The entry layer must control traffic spikes and policies while protecting the backend from a flood of requests.

2. Azure Functions as lightweight ingress

The HTTP trigger should accept, verify, persist, publish to Service Bus and respond, rather than trying to do everything.

3. Service Bus for retries and ordering

Service Bus sessions are a natural choice for ordered processing per entity. Duplicate detection provides additional protection but does not remove the need for idempotent workers.

4. Worker and domain logic

Whether using Functions, Container Apps or AKS, the principles matter most. Workers must handle retries, partial failures, redelivery and cases where a side effect happened before success was acknowledged.

5. Application Insights and DLQ

Without visibility from HTTP ingress through Service Bus to the business outcome, you are relying on hope rather than a stable webhook.

The central design principle

Keep expensive, long-running and side-effecting business work outside the HTTP request. Ingress accepts, verifies, persists, deduplicates, enqueues and acknowledges.

Idempotency: the foundation that makes retries useful

Because webhooks can arrive again, define an explicit idempotency model:

the provider event ID becomes the deduplication key,
ingress uses insert-if-absent in the accepted-event ledger,
only the first insert queues further work,
the worker performs side effects conditionally rather than blindly,
a repeat receives a technical response without repeating business effects.

That separates tolerating a retry from creating another order, email or fulfillment.

// pseudokod ingressu webhooka
verifySignature(rawBody, headers.signature, secret)

const providerEventId = headers["x-event-id"] || body.id
const dedupeKey = `${provider}:${providerEventId}`

const inserted = inboxStore.insertIfAbsent({
  dedupeKey,
  payloadHash: sha256(rawBody),
  receivedAt: now()
})

if (!inserted) {
  return http(200, { ok: true, duplicate: true })
}

queue.publish({
  dedupeKey,
  eventType: body.type,
  aggregateId: body.orderId || body.customerId,
  payloadRef: payloadStore.save(rawBody)
})

return http(202, { ok: true })

Event order: do not assume it unless you enforce it

“Webhooks will arrive in order” is a costly integration illusion. There is no automatic reason for that to happen.

For a stream such as:

ORDER_CREATED
ORDER_PAID
ORDER_CANCELLED

the system may see another ORDER_PAID, receive ORDER_CANCELLED before processing the previous event, or replay an older message after a retry.

If ordering matters, express it architecturally:

on AWS, use SQS FIFO + MessageGroupId = aggregateId,
on Azure, use Service Bus Sessions + SessionId = aggregateId.

The queue helps with technical ordering, but domain logic must still enforce business order. An older state must not overwrite a newer one simply because it arrived later.

Retries, backoff and DLQ: a policy is essential

Retrying is necessary but is not a complete strategy.

A mature approach does three things:

retries only errors that can reasonably be transient,
limits attempt count and event lifetime,
sends exhausted cases to dead-lettering with full context.

Otherwise, the system repeatedly pushes the same errors through the same architecture until downstream services suffer more.

A DLQ shows that the architecture does not pretend success where there was none.

Three realistic scenarios that expose weak architecture

1. A payment webhook arrives twice

The classic case: the provider sends payment.succeeded. The system records the transaction, updates the order and emails the customer. The HTTP response is slow or lost. The provider retries. Without idempotency, you may duplicate email, fulfillment or accounting entries.

Use the provider event ID as a deduplication key, insert it atomically if absent, start further work only on the first insert and perform worker side effects conditionally.

2. Logistics statuses arrive out of order

You have events: shipment.created, shipment.picked_up, shipment.in_transit, shipment.delivered. If delivered arrives early or an older event is replayed later, state becomes inconsistent. Deduplication alone is insufficient: use per-entity ordering and state-transition validation.

3. Outbound webhooks to unreliable customer endpoints

Your platform sends to hundreds or thousands of tenants. Some respond quickly; others return 429. Some take seconds, while others have unreliable 500. A shared dispatch path lets one problematic customer hurt everyone. Use tenant isolation, backpressure, retry policies, dead-lettering and hard connection timeouts.

What to watch for

Do not equate 2xx with business success.
Do not perform expensive logic within the webhook request.
Do not postpone idempotency.
Do not assume ordering you have not enforced.
Treat the DLQ as part of reliability architecture.
Log more than “received webhook”: event ID, tenant, aggregate, retry count and final outcome.
Do not let one unreliable receiver destabilize the entire outbound pipeline.

Observability: correlation replaces guesswork

During an incident, you should be able to answer:

was this the first receipt or a duplicate?
was the event in the DLQ?
how many retries occurred?
did ingress, the queue or the worker fail?
did technical success lead to business success?

Otherwise, the system is opaque rather than stable.

A minimum set of fields worth retaining in logs and metrics:

provider_event_id
dedupe_key
tenant_id
aggregate_id
signature_verified
ingress_latency_ms
queue_enqueue_latency_ms
worker_processing_latency_ms
retry_count
dead_letter_reason
final_business_outcome

The webhook architect's checklist

Does the event have a stable, unambiguous identifier?
Do you verify the signature and timestamp before proceeding?
Do you persist the event before side effects?
Do you use insert-if-absent or another idempotency-ledger model?
Does ingress respond quickly and predictably?
Is the worker safe under retries, replays and partial failures?
Is ordering per aggregate explicitly modeled?
Are retries bounded, with a DLQ afterward?
Can you perform controlled redrive from the DLQ?
Can you correlate the complete path from ingress to business effect?

Why it matters to the business

Weak webhooks cause duplicate payments, inconsistent orders, misleading customer communication, incorrect process starts, overloaded dependencies and technical debt that emerges as traffic grows.

Well-designed webhooks are predictable. That predictability directly reduces the cost of faulty automation.

A key question for the team

Is our webhook actually resilient to duplicates, retries, timeouts and bursts, or have we simply not encountered the conditions that expose its weaknesses?

Ultimately, the whole flow matters

Webhook stability belongs to the complete flow: request acceptance, ledger, deduplication, queue, worker, retry, dead-lettering and final business effect.

The best webhooks are boring, predictable and resilient. That is exactly what they should be.

At ArcCloud, we view integrations as operational decisions: what was accepted, what was acknowledged, what needs retrying, what reached the DLQ and where intervention is required. System stability starts when guesswork ends.