GraphQL Query Scraping: Read, Replay, and Know When to Stop

What a GraphQL Query Actually Is (And Why It Shows Up in Scraping)

A GraphQL query is a request written in a query language for APIs that asks for exactly the fields a client needs, in one round trip, instead of hitting several fixed endpoints the way a REST API does. Where REST typically exposes separate URLs like /users/1 and /users/1/orders, GraphQL exposes a single endpoint — almost always /graphql — and lets the client describe the shape of the data it wants inside the request body itself.

This is why scraping engineers keep running into GraphQL instead of clean HTML. When you view-source a modern site built with React, Vue, or a similar framework, the page is often mostly empty markup. The real content — product listings, comment threads, search results — loads afterward via a JavaScript fetch to that /graphql endpoint, using data returned from a server like Apollo Server or Hasura. The graphql query vs rest api distinction matters here: with REST you can often guess a URL pattern; with GraphQL, everything funnels through one POST request, and the actual "route" is encoded inside the JSON payload rather than the URL.

Finding the Query: Reading the Network Tab

To find a graphql query in the network tab, open your browser's DevTools, switch to the Network panel, filter by Fetch/XHR, and reload the page. Look for a POST request to a path ending in /graphql (or sometimes /api/graphql, /query). Click it, then inspect the request payload.

You'll typically see three properties in the JSON body: query, variables, and often operationName. The query property holds the actual GraphQL document — the field selections the client is asking for. variables holds the dynamic inputs, like a page number or search term. operationName labels the operation, useful when a client bundles multiple queries together. Learning to intercept a GraphQL request this way is the same technique described in Apify Academy's guide to GraphQL scraping, which walks through identifying the endpoint and payload structure in more detail. Once you've isolated the query and variables, you have, in principle, everything needed to replay that exact request outside the browser.

Query Variables and Pagination

Variables let a single query definition serve many different requests. Instead of hardcoding a page number into the query text, the query declares a placeholder — say $cursor: String — and the variables object supplies the value at request time. A simplified graphql query example for paginated results looks like this:

query ProductList($cursor: String, $limit: Int) {
  products(after: $cursor, first: $limit) {
    edges { node { id name price } }
    pageInfo { endCursor hasNextPage }
  }
}

Here, pageInfo.endCursor from one response becomes the $cursor variable for the next request — a graphql pagination cursor pattern common in Apollo- and Relay-based frontends. To build a repeatable extraction loop, you don't need to touch the query text at all; just keep updating variables and checking hasNextPage until it returns false. This is also where fragments often appear — reusable field selections referenced inside the query — so don't be surprised if the full document includes a fragment block alongside the operation itself.

Introspection: When You Can (and Can't) See the Schema

GraphQL introspection is a built-in feature that lets a client ask the API to describe its own schema — every type, field, and argument available. In development it's usually on, which makes graphql schema discovery trivial with tools like GraphiQL. In production, though, introspection is commonly disabled for security reasons, as GraphQL.org's own documentation confirms — exposing your full schema to anyone who asks makes it easier to find unused or sensitive fields.

When introspection is disabled in production, you're not entirely blind. Many servers still return field-suggestion errors — "did you mean products?" — when a client sends an almost-correct field name. As PortSwigger's Web Security Academy explains, these suggestion messages can leak enough of the schema to reconstruct fields through trial and error, even with introspection turned off. It's slower than reading a schema directly, but it's a real path when the front door is locked.

Where Query Replay Breaks Down

Copying a captured query into a script and getting it to work once is easy. Getting it to keep working is where reverse-engineering a GraphQL API gets fragile. Four failure points show up repeatedly:

• Persisted query hashes. Many clients send only a SHA-256 hash of the query, not the query text itself, via the persistedQuery extension. Replay it without the hash the server expects and you'll get a PersistedQueryNotFound error — a scenario broken down in detail in Crawlee's reverse-engineering write-up on persisted query hashes.
• Rotating auth headers. Bearer tokens, CSRF tokens, or session cookies tied to the query often expire or rotate per session, so a query that worked yesterday returns a 401 today.
• Query-complexity limits. Some servers reject deeply nested or high-volume queries outright, throttling anything that looks automated.
• Bot detection on the endpoint itself. The /graphql route can sit behind the same fingerprinting and rate-limiting as the rest of the site, and a graphql query blocked mid-scrape is often a bot-detection signal rather than a schema problem — worth diagnosing with Bot Detection Headless Browser: How to Spot a Block.

When to Render the Page Instead of Replaying the Query

Direct query replay is the right call when the payload is stable, auth is simple (or absent), and the hash — if there is one — doesn't rotate unpredictably. It's fast, cheap on compute, and easy to maintain.

Rendering the page becomes the more resilient option once any of that breaks down: rotating persisted-query hashes you can't reliably predict, session-bound tokens generated client-side, or bot detection guarding the API but not the rendered page. A headless browser loads the page the way a real user's browser would, executes the same JavaScript, and lets you capture the resulting DOM or intercept the network calls without needing to reverse-engineer auth logic yourself. It's slower per request than a raw GraphQL call, but it doesn't break the moment the frontend team rotates a hash.

Browsevra's docs walk through exactly this: rendering a JavaScript-heavy page and pulling structured data or screenshots straight from the loaded DOM, bypassing the need to replay the query at all. For teams running this at scale, pricing details what that looks like in production volume.

Query replay is the fast path when the API cooperates; rendering is the safety net when the site fences it off. Rather than spending engineering hours chasing rotating hashes and auth walls, point your pipeline at the rendered page and let browsevra handle the JavaScript execution for you.

Frequently Asked Questions

Is scraping a GraphQL API legal if the endpoint is public?

Legality depends on jurisdiction, the site's terms of service, and what data you're collecting — this isn't legal advice, but publicly accessible doesn't automatically mean scraping is permitted. Review the target's terms of service and any applicable data protection laws before building a scraper, and avoid collecting personal or authentication-gated data without permission.

Why does my GraphQL query work in the browser but fail when I replay it with curl or requests?

It usually fails because the browser sends additional context your script doesn't — cookies, CSRF tokens, a persisted-query hash, or headers like Referer and Origin that the server checks. Copy the full request headers from DevTools, not just the JSON body, and replicate them in your Python requests call.

What's the difference between scraping a REST API and a GraphQL query?

REST typically spreads data across multiple fixed URLs, so scraping means hitting several endpoints and stitching responses together. GraphQL concentrates everything into one endpoint and one POST body, where the query and variables fields define exactly what's returned — fewer endpoints to find, but a payload structure to decode instead.

Can I still get the schema if GraphQL introspection is disabled?

Sometimes — field-suggestion error messages ("did you mean...?") can leak partial schema information even with introspection off, as documented by PortSwigger's security research. It's slower than reading an introspected schema directly, requiring trial-and-error field guessing based on the site's known frontend behavior.

Why did my GraphQL scraper suddenly start returning a PersistedQueryNotFound error?

This happens when the client sends only a hash of the query instead of the full query text, and the server doesn't recognize that hash — often because it rotated after a frontend deploy. You need to either capture the current hash fresh from the network tab or send the full query text with the persistedQuery extension included, as Crawlee's case study explains.

When should I just render the page instead of replaying the GraphQL query directly?

Switch to rendering once auth tokens rotate per session, persisted-query hashes change unpredictably, or the /graphql endpoint shows signs of bot detection. A headless browser executes the page's JavaScript like a real user, capturing the same data through the DOM without needing to maintain fragile query-replay logic.