Webhook API Async Job Patterns for Headless Browser Tasks

Why Polling Breaks Down for Long-Running Browser Jobs

Rendering a heavy single-page app, generating a multi-page PDF, or scraping a site behind a queue doesn't take a fixed amount of time. It might finish in 2 seconds or take 2 minutes if the page is JavaScript-heavy, the queue is backed up, or the target site is slow. This variability makes polling vs webhook a real architectural decision, not a matter of taste.

A polling-based integration hits a job status endpoint every second (or more often) until it sees a "completed" state. At small scale this feels harmless. At production scale — hundreds or thousands of concurrent jobs — it becomes a liability: you're opening and tearing down connections repeatedly, burning API quota on requests that return "still processing" 95% of the time, and adding latency because your poll interval, not actual job completion, determines how fast you find out the result is ready.

A headless browser API is a particularly bad fit for polling because job duration is inherently unpredictable. A static page screenshot and a 40-page PDF export from a dynamic dashboard don't take the same time, and there's no poll interval that's efficient for both. Webhooks solve this by inverting the model: instead of asking "is it done yet?" on a timer, you get told the moment it is.

The Async Callback Pattern: How It Works

The async callback API pattern is straightforward once you see the full request/response contract:

1. Your client submits a job — a screenshot, PDF render, or scrape request — and includes a callback_url parameter pointing at your own endpoint.
2. The API immediately responds with 202 Accepted and a job ID. No rendering has happened yet; this just confirms the job was queued.
3. The headless browser executes the job asynchronously, on its own schedule, unconstrained by your client waiting on an open connection.
4. On completion (or failure), the API sends a signed HTTP POST to your callback_url containing the job status, result URLs (e.g., the rendered screenshot or PDF file), and metadata like timing and page info.

This is a job-completion callback, not a general-purpose event bus — you're not subscribing to a stream of arbitrary events, you're asking to be notified once when a specific job finishes. That distinction matters for the receiver design: it only needs to handle a small, predictable set of payload shapes tied to job IDs you issued yourself.

Designing Your Webhook Receiver

Building a solid webhook receiver comes down to three disciplines.

Respond fast, then process. Your endpoint should return a 2xx status within a few seconds of receiving the POST — acknowledge receipt, then hand off actual processing (saving the file, updating your database, notifying users) to a background worker. If your endpoint takes too long, the sender may interpret that as a failure and retry, creating duplicate work.

Verify the signature before trusting anything. Any endpoint accepting POST requests from the public internet is a target for spoofing. Webhook signature verification using HMAC-SHA256 is the industry-standard defense: the sender computes a hash of the payload using a shared secret, includes it in a header, and you recompute that hash before trusting the body. If implementing this from scratch, this guide to HMAC webhook signatures walks through the mechanics, and the Standard Webhooks specification documents the conventions most providers converge on for signing and header formats.

Treat every delivery as potentially duplicate. Networks retry, timeouts happen, and a well-designed webhook system will occasionally send the same completion notification twice. Use the job ID as an idempotency key — before acting on a payload, check whether you've already processed that job ID, and discard the duplicate rather than reprocessing it. This single habit eliminates most pain around duplicate or out-of-order deliveries.

Handling Retries, Failures, and Timeouts

Webhook delivery isn't guaranteed to succeed on the first attempt — your server might be deploying, restarting, or down. Production-grade webhook retry logic uses exponential backoff: after a failed attempt, the sender waits a short interval, then doubles it on each subsequent failure, spacing retries over minutes or hours rather than hammering a dead endpoint.

Eventually, retries have to stop. Deliveries that exhaust all attempts should land in a dead-letter queue — a durable record of failed notifications you can inspect and manually reconcile rather than silently losing.

This is also why a fallback polling endpoint should never be removed entirely, even in a webhook-first design. If your server was down for the entire retry window and missed every attempt, a lightweight job status endpoint lets you reconcile state after the fact. Webhooks handle the common case efficiently; polling handles the edge case safely.

Webhooks + Polling: A Hybrid Pattern That Actually Works

Most production systems don't pick one exclusively — they run a hybrid webhook polling architecture. The webhook is the primary notification channel, driving real-time updates the moment a job finishes. A job status endpoint sits alongside it as a backup, used sparingly for reconciliation: checking jobs where no webhook arrived within an expected window, recovering from an outage, or auditing that your webhook processing didn't silently drop anything.

This pairing gives you the efficiency of push-based notification with the safety net of pull-based verification, without forcing you to poll every job on a tight interval as your default behavior.

Implementing Callbacks with Browsevra

Browsevra's async job API applies this pattern directly. Pass a callback_url parameter when you submit a screenshot, PDF, or scraping job, and Browsevra returns a job ID immediately with a 202 response while the render runs in the background. Once complete, Browsevra POSTs a signed payload to your callback URL with the job status, result file URLs, and metadata — no polling loop required for the common path.

This applies across job types with different duration profiles: PDF generation (see the pixel-perfect HTML to PDF guide), SPA scraping that depends on JavaScript execution finishing (covered in rendering JavaScript for SPAs reliably), and high-concurrency screenshot workloads that need queue-aware handling (detailed in the concurrency and queue playbook). For the exact callback payload schema and signature header format, the Browsevra docs are the authoritative reference.

If you're still polling a job status endpoint in a loop, it's worth wiring up a callback instead — check the docs for the exact parameters, review pricing for job-based usage tiers, and get your first webhook-driven job running in minutes with browsevra.

Frequently Asked Questions

Why shouldn't I just poll a job status endpoint every second?

Polling every second wastes API quota and connection overhead on requests that mostly return "still processing," and caps your responsiveness to whatever interval you chose rather than actual completion time. At scale — thousands of concurrent jobs — this adds meaningful load on both your infrastructure and the API provider's. Webhooks eliminate the wasted requests by notifying you exactly once, when the job is actually done.

How do I verify that a webhook request actually came from the API provider?

Verify the HMAC-SHA256 signature in the request headers by recomputing it against the raw payload using your shared secret, then comparing it to the provided value. If the signatures don't match, reject the request as potentially forged. This is the same mechanism described in the Standard Webhooks specification and used across most production webhook systems.

What happens if my server is down when the webhook is sent?

The sender retries delivery using an exponential backoff schedule, spacing attempts further apart over time. If all retries are exhausted, the delivery typically lands in a dead-letter queue for manual review, which is why a fallback polling endpoint should remain available to reconcile any jobs you might have missed entirely.

Can I use both webhooks and polling for the same job?

Yes, and this hybrid approach is what most production systems actually run. Webhooks serve as the primary, low-latency notification channel, while a lightweight status endpoint acts as a backup for reconciliation when a webhook doesn't arrive within an expected window.

What's the difference between a webhook and a callback URL?

A callback URL is the specific endpoint you provide on a single job request, telling the API where to send that job's completion notification. A webhook is the actual HTTP POST delivered to that URL when the event occurs — the callback URL is the destination, the webhook is the message.

How long should I wait before considering a rendering job failed if no webhook arrives?

Set a timeout based on your job type's expected duration range plus a margin for retry delays — often a few minutes for simple renders and longer for heavy SPA or multi-page PDF jobs. Once that window passes with no webhook, query the fallback job status endpoint directly rather than waiting indefinitely.