Web Scraping Legality: A Decision Framework for Engineers

There's No Single 'Web Scraping Law'

Ask "is web scraping legal?" and the honest answer is: it depends. Legality isn't governed by one statute — it sits at the intersection of contract law (terms of service), computer-crime statutes like the CFAA, copyright, and privacy regimes like GDPR, each of which can independently create liability.

You don't need another summary of hiQ Labs v. LinkedIn or Meta v. Bright Data — we've covered that in our case-law deep dive and in a developer-focused risk guide. What you need is a repeatable framework to apply before you write a line of code.

The Four Factors That Actually Determine Legality

Every scraping project's legal risk can be assessed along four independent axes. None alone is dispositive — a project can be low-risk on three and high-risk on the fourth — but together they give a defensible, structured answer. Treat this as the core of your web scraping compliance process.

What you scrape: public data vs. personal or protected data

Scraping publicly visible, non-personal data — prices, inventory counts, aggregate statistics — is broadly lowest-risk. Personal data (names, emails, profile details, anything tied to an identifiable person) is different, because GDPR attaches obligations to personal data regardless of whether it was publicly accessible. "The data was public" is a common but weak defense; CNIL and other European regulators have made clear that public availability doesn't remove the requirement for a lawful basis to process that data, per current GDPR compliance guidance. Copyrighted or proprietary content (article text, curated databases, images) adds a separate layer of risk.

How you access it: public pages vs. bypassing barriers

The method of access matters as much as the content. Requesting a public page like a browser normally would differs, legally, from logging in with credentials you weren't issued, bypassing a paywall, or defeating a CAPTCHA. This maps to the "authorization" logic in CFAA-style statutes: courts have generally been skeptical of claims against scraping freely accessible pages, but circumventing an access control — even a weak one — reintroduces the unauthorized-access argument that sank early scraping defenses. If your infrastructure is getting flagged or throttled, read how bot detection actually works — a block often signals the site considers your access non-compliant.

Where you and the data subjects are: jurisdiction

Jurisdiction isn't about where your servers sit. GDPR applies based on where data subjects are located, not where the scraper operates — a US company scraping EU residents' profiles is squarely in scope, as detailed in DataHen's analysis of GDPR and enterprise scraping. On the US side, CFAA interpretation and state-level computer-crime law vary by circuit, documented in McCarthy's state-by-state and circuit-split guide. A project fine under one circuit's precedent may not be under another's, and being a US company doesn't shield you from EU exposure if EU residents' data is involved.

What you do with it: the use case matters

Downstream use independently raises or lowers scrutiny. Internal price monitoring or academic research sits at the low end. Republishing scraped content, building a competing product on someone else's database, or feeding scraped data into AI model training sits at the high end — the latter is under active litigation, with disputes like Reddit v. Perplexity testing how far "publicly available" data can be repurposed for AI training without consent or licensing. If your output is a training set, resold dataset, or public-facing derivative, treat that as a separate risk factor from the scrape itself.

Common Mistakes That Turn Legal Scraping Into a Liability

Most scraping disputes start with avoidable mistakes, not a clean architecture gone wrong. Ignoring robots.txt is one: it has no independent legal force, but ignoring it after a site operator references it in a cease-and-desist, or alongside ToS language, has been used as evidence of intent — treat robots.txt as part of your risk picture even though it's not a statute. Hammering endpoints without rate limits invites load-based claims resembling trespass-to-chattels arguments. Scraping behind a login wall turns a public-data question into an authorization question. Storing personal data without a lawful basis under GDPR creates liability independent of how the data was collected. And buying a pre-scraped dataset without checking provenance just inherits someone else's risk — you're liable for what's in it, not just what you scraped yourself.

A Practical Compliance Checklist for Your Scraping Stack

Before building, read the target site's terms of service and robots.txt, and record what they said at the time you accessed the site. Rate-limit and cache aggressively — this undercuts load-based legal theories and reduces operational risk simultaneously. Never bypass logins, paywalls, or CAPTCHAs as standard practice; treat that as an escalation trigger, not a technical challenge. Minimize personal data collection to what you actually need, and document a lawful basis wherever GDPR or similar law applies. Log your sources, access dates, and the ToS/robots.txt state at scrape time — that documentation turns "we think we were compliant" into evidence. Escalate to legal counsel when: the target contains personal data at scale, the use case is AI training or resale, you're operating across multiple jurisdictions, or you're considering any access-control bypass. Standard engineering safeguards handle the rest. For a fuller checklist framed around risk tiers, see our developer's risk guide; if your method is UI-based rather than API-based, considerations shift slightly — covered in our screen-scraping piece. If you're still evaluating what tool fits your pipeline, our tool category map is a useful next step.

Much of this comes down to technical controls, not just policy: rate limiting, robots.txt handling, session management, and avoiding credential bypass are engineering choices that directly shape legal exposure. A managed headless browser API can enforce many of these by design rather than leaving them to individual scripts. Browsevra's docs show how request pacing and session handling work in practice — worth a look before your next scraping project goes from prototype to production at browsevra.

Frequently Asked Questions

Is web scraping illegal?

Not inherently — scraping publicly accessible, non-personal data is generally lower risk and has survived challenges like hiQ v. LinkedIn. Risk rises based on how you access the data (bypassing logins or CAPTCHAs), what you scrape (personal or copyrighted content), and what you do with it (resale, republishing, AI training). There's no blanket answer; it depends on the specific combination of factors.

Does ignoring a website's robots.txt file break the law?

Robots.txt itself isn't a law and has no independent statutory force. However, ignoring it can be used as evidence of intent in disputes over authorization or contract violation, especially if a site owner has referenced it in a cease-and-desist. Treat it as a meaningful signal, not a legal requirement in isolation.

If data is public, can I still violate GDPR by scraping it?

Yes. GDPR attaches obligations to personal data based on whether it identifies a person, not on whether it was publicly visible. Regulators including CNIL have confirmed that public availability doesn't remove the need for a lawful basis to collect and process that data.

Can a website's terms of service alone make scraping illegal?

ToS violations are primarily a contract-law issue, creating civil liability, but they don't automatically constitute a criminal offense under statutes like the CFAA. Combined with access-control circumvention or bad-faith use of the data, though, ToS violations can strengthen a plaintiff's broader legal claim.

Is it legal to scrape websites to train an AI model?

It's legally riskier than internal or research use, and currently a live area of litigation — disputes like Reddit v. Perplexity are actively testing the boundaries. AI training use cases face more scrutiny around consent, licensing, and downstream commercial use than lower-profile internal analytics projects.

Do I need a lawyer to review a scraping project before I build it?

Escalate to legal counsel when the target involves personal data at scale, cross-border data subjects, AI training or resale use cases, or any access-control bypass. For lower-risk projects — public, non-personal data collected respectfully for internal use — standard engineering safeguards are typically sufficient.