Shadow DOM Scraping: How to Extract Data From Web Components

Shadow DOM scraping is the practice of extracting data from web pages that hide their markup inside encapsulated custom elements — and it trips up almost every scraper not built to handle it. If you've pointed a scraper at a page, watched it render perfectly in a browser tab, then gotten null or an empty string back from your script, shadow DOM is very likely the reason.

This isn't the usual "shadow DOM for testing" explainer aimed at QA engineers writing Playwright assertions. It's a practical guide for developers who need to pull structured data out of pages built with modern design systems, reliably at scale — not just once, in a debugger.

Why Shadow DOM Breaks Ordinary Scrapers

Shadow DOM is a browser feature that lets a custom element attach its own isolated DOM subtree — a shadow root — with markup and styles that don't leak into, or get affected by, the rest of the page. It's the mechanism behind design systems like Shoelace, Spectrum, and countless internal component libraries. From a UI engineering standpoint, that encapsulation is the whole point.

From a scraping standpoint, it's a wall. document.querySelector stops at the boundary of a shadow root unless you explicitly walk into it. Raw HTML parsers — BeautifulSoup, Cheerio, or a requests call followed by regex — never see shadow content at all, because they only operate on the HTML fetched over the wire. If a page assembles its content via JavaScript that attaches shadow roots after load, the initial HTML response contains none of it: no text, no attributes, just empty custom element tags waiting to be hydrated.

This is why the scraper returns null even though a human looking at the rendered page sees the data plainly. The content exists in the composed DOM the browser builds at runtime, not in the document source. Any pipeline that treats "fetch the page" and "parse the HTML" as separate, static steps will silently miss everything inside encapsulated markup — no error, no exception, just missing fields where a screenshot clearly shows a price, a review, or a table.

Open vs. Closed Shadow Roots: What You Can and Can't Reach

The practical distinction for extraction is whether a shadow root was attached with mode: 'open' or mode: 'closed'.

An open shadow root exposes element.shadowRoot, so any script running in the page — including code injected via a headless browser — can traverse into it just like a normal DOM node. Most component libraries use open roots by default, since closing them off makes debugging and third-party integration painful, so most shadow DOM scraping problems are solvable with straightforward traversal.

Closed shadow roots are different by design: element.shadowRoot returns null, deliberately blocking script-level access even from code running in the same page context. There's no supported DOM API workaround. The realistic paths are either accepting the data isn't accessible through page-level scripting, or using Chrome DevTools Protocol-level access from a controlled browser session, which can inspect the render tree beneath the script sandbox. That CDP route is meaningfully more complex and worth reserving for cases where open-root traversal genuinely isn't an option — as this deep dive on Playwright's shadow DOM mechanics explains, querySelector simply stops at a closed boundary with no elegant workaround from standard scripting.

A Working Pattern for Piercing Nested Shadow Trees

For open shadow roots — the common case — a recursive walk handles arbitrary nesting depth. Run something like this inside page.evaluate():

function extractShadowData(root = document) {
  const results = [];
  const walk = (node) => {
    const all = node.querySelectorAll('*');
    all.forEach((el) => {
      if (el.shadowRoot) {
        results.push({
          tag: el.tagName.toLowerCase(),
          text: el.shadowRoot.textContent?.trim(),
        });
        walk(el.shadowRoot); // recurse into nested shadow trees
      }
    });
  };
  walk(root);
  return results;
}

This checks every element for a live shadowRoot, records what it finds, then recurses into that shadow root looking for further nested custom elements — because design systems frequently nest components (a containing a containing an , each with its own shadow tree). Swap textContent for targeted attribute reads or nested querySelector calls once you know the structure you're after.

If you'd rather not write a manual walker, modern Playwright versions pierce open shadow DOM automatically in their CSS selector engine — a selector like page.locator('my-component >>> .price') reaches into nested open shadow roots without extra code, as documented in Playwright's element selectors reference. That's often the fastest path when extracting a handful of known fields rather than crawling an unknown component tree.

Declarative Shadow DOM: A New Wrinkle for Scrapers

Declarative shadow DOM changes one of the assumptions above. Using