How to Render JavaScript for Scraping SPAs Reliably

and a pile of tags. Everything a user sees gets injected afterward, client-side, by JavaScript that curl never executes. For the full diagnosis, see Dynamic Content Scraping: Why Pages Load Blank & How to Fix. The short version: to scrape a single page application, you need a real browser engine running the page's JavaScript, plus a rendering API or headless browser instance that knows when to stop waiting and hand you the DOM. That "when" is where most scrapers break, and it's what the rest of this guide solves.

Strategy 1: Waiting for Network Idle

The most common first attempt is telling the browser to wait until the network goes quiet. Puppeteer and Playwright expose this as a waitUntil option with two relevant values: networkidle0 waits until there are zero active network connections for at least 500ms, while networkidle2 waits until there are no more than two active connections for that window. As BrowserStack's guide to Puppeteer's waitUntil explains, networkidle2 is more forgiving — it tolerates a lingering analytics beacon or slow-polling connection, whereas networkidle0 demands total silence.

await page.goto(url, { waitUntil: 'networkidle2' });

Use networkidle0 for simple, mostly-static pages where you want maximum certainty everything has finished loading. Use networkidle2 for typical SPAs with third-party scripts, ads, or chat widgets that never fully stop talking to the network.

The classic failure mode is a page that never goes idle: dashboards with websocket connections, live-updating widgets, or infinite background polling keep at least one connection open indefinitely. Wait for network idle on one of these pages and your request times out — not because the content isn't ready, but because the idle condition can never be satisfied. This is the first sign you need a more targeted wait strategy.

Strategy 2: Waiting for a Specific DOM Selector

Rather than waiting for the network to behave, you can wait for the content you care about to appear. waitForSelector polls the DOM until a given selector exists (or is visible), then resolves immediately — often faster than any network-based wait, and immune to background polling that never quiets down. Autify's guide to waitForSelector in Playwright confirms this is the current recommended approach for content-driven waits rather than arbitrary timeouts.

await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('[data-testid="product-price"]', { timeout: 10000 });

Selector choice matters as much as the technique itself. CSS classes tied to styling frameworks change on every deploy; data-testid or other semantic attributes are far more stable because teams treat them as contracts, not styling hooks. Selector logic built on brittle class names is a maintenance trap — it works today and breaks silently the next time a frontend team ships a redesign.

The failure mode to watch for: a selector that exists in the DOM but is empty, hidden, or a skeleton placeholder. waitForSelector only confirms presence, not that real data has loaded into it — the gap the next strategy closes.

Strategy 3: Custom JS Execution for Conditional Waits

Some loading conditions aren't expressible as a selector at all — a loading spinner that needs to disappear, a global JS variable that flips to true once an API response lands, or content that only appears after you trigger infinite scroll or click a "load more" button. For these, page.evaluate() lets you run arbitrary JavaScript inside the page context and poll until a condition is true.

await page.waitForFunction(() => {
  return window.__APP_STATE__?.dataLoaded === true;
}, { timeout: 15000 });

// Trigger a "load more" click before extracting
await page.evaluate(() => {
  document.querySelector('.load-more-btn')?.click();
});

This is custom JS execution at its most powerful: instead of guessing at network behavior or DOM presence, you're directly querying the page's internal state. It's also the right tool for lazy-loaded images and infinite-scroll feeds, where you may need to scroll, wait, and re-check in a loop before the full dataset exists in the DOM. The tradeoff is complexity — this requires knowing something about the target site's internals, and that logic can break silently when the site's frontend code changes.

Combining Strategies for Reliable Extraction

Production scrapers rarely rely on one strategy alone. A pattern that survives real traffic: navigate with domcontentloaded (fast, doesn't wait on the network at all), then waitForSelector for the primary content block with a reasonable timeout, then run a short custom JS check or scroll/click trigger to handle lazy content, then extract. If the selector wait times out, fall back to a fixed grace period rather than failing the whole job outright. This layered approach is what it actually means to render JavaScript for scraping at scale — no single wait condition covers every page state, but stacking them catches what the others miss. Pair this with a suitable library, as discussed in Python Scraping Tools: The Layer-by-Layer Decision Guide, and you have a resilient extraction pipeline.

Doing This Without Managing Browsers Yourself

Every strategy above requires provisioning, updating, and babysitting real browser instances — a real operational cost. A managed JavaScript rendering API exposes these same wait strategies as simple request parameters instead of browser automation code. For background on how these APIs work generally, see Headless Chrome API: What It Means and How to Use One.

With Browsevra, wait_until, wait_for_selector, and custom JS execution are just fields in a JSON request:

{
  "url": "https://example.com/product",
  "wait_until": "networkidle2",
  "wait_for_selector": "[data-testid=\"product-price\"]",
  "js_execution": "document.querySelector('.load-more-btn')?.click();"
}

No browser pool to maintain, no timeout tuning across a fleet of Chromium instances — you send the same parameters described in this guide and get back rendered HTML, a screenshot, or a PDF. Read the Docs for the full parameter reference, check Pricing to see what fits your volume, or explore browsevra to try it against your own SPA target today.

Frequently Asked Questions

Why does my headless browser screenshot show a loading spinner instead of the actual content?

Your wait condition resolved before the real content finished rendering — usually because waitUntil: 'load' or a short network idle check fired while a spinner was still on screen. Switch to waitForSelector targeting the actual content element, or use page.evaluate() to poll until the spinner is removed from the DOM.

Is waitForSelector better than a fixed delay like sleep(5) for scraping SPAs?

Yes — waitForSelector resolves as soon as content is actually present, making it faster on quick-loading pages and safer on slow ones, where a fixed 5-second sleep might not be enough. A hardcoded delay is a guess; a selector wait is a direct check against reality.

Can I wait for a JavaScript variable or API response instead of a DOM element?

Yes, using page.waitForFunction() you can poll any expression evaluated in the page context, such as a global state variable flipping to true once an API call resolves. This is useful when the data you need isn't reflected in the DOM immediately, or when a component renders before its data has arrived.

What happens if a page never reaches network idle because of ads or analytics scripts?

The wait will time out, since networkidle0 and networkidle2 both require the network to go quiet for a sustained window that persistent background scripts prevent. The fix is to abandon network-based waiting for that page and switch to waitForSelector or a custom JS condition tied to the content itself.

Do I need Puppeteer or Playwright, or can a rendering API handle waiting for me?

A rendering API can handle all of this for you, exposing wait_until, wait_for_selector, and custom JS execution as simple request parameters instead of code you write and maintain. This removes the need to run and update Puppeteer or Playwright yourself while still giving you full control over the same wait strategies.