A webhook can accept an event in milliseconds and still lose the business action behind it. Learn when to add a durable queue, safe retries, idempotency, and manual replay.
A webhook queue becomes necessary when receiving an event and completing the business process can no longer be treated as one reliable operation. A form submission may arrive instantly, but the workflow behind it still depends on a CRM, an email provider, an accounting system, or an AI service. Any of those systems can slow down, reject a request, or become unavailable after the sender has already moved on.
The common first version handles everything inside the incoming request: validate the payload, look up a contact, call several APIs, update records, and finally return a success response. That is acceptable for a low-volume, low-consequence workflow. It becomes fragile when the event matters enough that losing it would require a customer apology, a manual reconciliation, or a financial correction.
A webhook sender and your workflow have different responsibilities. The sender wants a quick answer confirming that the event was received. Your business process may need much longer to finish. Combining those two lifecycles means one slow dependency can make the sender believe delivery failed, even if part of your workflow already ran.
That creates an awkward failure mode. Imagine an illustrative order workflow that creates a customer record, then times out while calling the fulfillment system. The sender retries because it never received a timely response. The second execution may create another customer, another task, or another shipment unless the workflow recognizes the event. Retrying the whole scenario is not recovery if earlier steps already produced side effects.
More timeout settings do not solve this. They only extend how long the receiving endpoint remains tied to downstream systems it does not control. Once an integration crosses several APIs or includes variable-duration work such as document extraction or model inference, the request-response path is the wrong place to finish the job.
The safer design has two stages. The intake endpoint authenticates the request, performs minimal schema validation, records the event in durable storage, and returns a successful response. A separate worker then claims the stored event and performs the actual business actions.
Minimal does not mean careless. The intake stage should reject invalid authentication and malformed payloads rather than preserving junk. It should also capture enough context to investigate later: the provider event ID, event type, received timestamp, relevant headers, raw or normalized payload, and the integration version that accepted it.
Do not place optional enrichment in this path. A CRM lookup, contact merge, AI classification, or outbound notification belongs in processing. The intake service should remain available even when those dependencies are not.
For a modest operation, durable storage does not require elaborate infrastructure. A database table with controlled claiming can be enough. A managed queue becomes useful when you need higher throughput, independent workers, delayed retries, or stronger concurrency controls. The important decision is not the logo on the tool. It is whether an accepted event survives a worker crash and remains visible until it reaches a final state.
Our take is that automatic retries are dangerous until the workflow is idempotent. Teams often enable retry-on-error first because it feels like a reliability feature. In practice, it can turn a temporary API problem into duplicate invoices, duplicate messages, or repeated pipeline changes.
Every event needs a stable idempotency key. Prefer the source provider's event ID when it is documented as unique. If none exists, derive a key from stable business fields and document the collision risk. Store that key before executing side effects, and enforce uniqueness at the storage layer rather than relying on a visual workflow branch that two concurrent runs can both pass.
Each downstream action also needs a known duplicate strategy. Some APIs accept their own idempotency key. For others, store the resulting record ID immediately after creation and check it before another attempt. A retry should resume from a known checkpoint where possible, not blindly replay the workflow from step one.
Classify failures before scheduling another attempt. Timeouts, connection failures, rate limits, and many server errors may recover. Invalid credentials, rejected payloads, missing required data, and permission errors normally need intervention. Repeating a permanent failure every five minutes adds noise and may trigger provider limits without moving the event closer to completion.
A queue is not complete when it can retry. It is complete when an operator can answer four questions: what is waiting, what failed, what has already changed, and what action is safe now?
Use explicit states such as received, processing, retry scheduled, completed, and needs review. Record the attempt count, next attempt time, last error category, downstream response reference, and the identifier of every record created. Keep a final failure bucket for events that exhaust their retry budget or require a business decision.
The replay control should be narrow. An operator should be able to retry one event after fixing its data or credentials without rerunning every successful event in the batch. If processing can partially succeed, expose the completed checkpoint and the next intended action. A generic “run again” button hides the most important information.
Alerts should reflect consequence, not every transient error. One temporary timeout that recovers on the next attempt is useful operational data, but it may not deserve a page. A growing queue age, repeated authentication failures, or an event that reaches the final failure state needs attention. Monitor oldest unprocessed event, completion latency, retry volume, final failures, and duplicate attempts blocked.
There is no universal event-volume threshold. A workflow processing ten high-value payment events per day may deserve durable handling before a marketing sync processing thousands of replaceable updates. We decide based on consequence and execution variability.
Add a queue when an accepted event must not disappear, when processing spans multiple external systems, when the sender may retry, when execution time varies widely, or when you need controlled concurrency against a downstream API. Keep the synchronous design when the work is short, easily repeated, low consequence, and handled by a platform whose delivery and recovery behavior you have actually verified.
A reasonable first production version has one intake endpoint, one event store, one worker, a stable idempotency key, bounded retries, and a manual review state. It does not need a fleet of services. Overbuilding the infrastructure before the workflow has real load creates another system to operate. Underbuilding the recovery path creates hidden operational debt.
Over the next 6 to 12 months, our view is that queue depth, idempotency, replay, and event-level tracing will become more visible inside automation platforms. AI steps make execution time and failure shape less predictable, so a single linear run history is no longer enough for critical workflows.
We also expect teams to separate workflow logic from delivery guarantees more deliberately. A no-code scenario may still orchestrate the business process, while a thin intake layer preserves events and controls retries. That hybrid pattern keeps iteration fast without asking the visual workflow to act as a durable message broker.
PASMO designs automation around the failure path as well as the successful run. If a webhook workflow has become important enough that “just rerun it” is no longer a safe answer, the intake, idempotency, and recovery model are the right places to start.