How to Scrape Without Getting Blocked: Recognize the Block, Then Fix It
Most blocking problems are fixed in the wrong order. A scraper fails, and the first reaction is to add proxies, switch to a headless browser or slow everything down, often all at once. It is faster to first work out which kind of protection stopped you, because each kind responds to a different fix, and some of them don't respond to proxies at all.
This guide shows how to read a block, what each kind usually needs, which proxies fit which job, and the lessons we learned keeping our own scrapers running. If you'd rather not deal with any of this, the scrapers in our catalogue handle it for the sites they cover.
Read the block first
Look at the status code, the response headers and the first lines of the body before changing anything. These are the patterns we run into most:
| What you see | Usually means | What tends to help |
|---|---|---|
HTTP 429, sometimes with a Retry-After header | A plain rate limit | Slow down, respect Retry-After, spread requests over more IPs |
| Works for a few dozen requests, then every request fails from that IP | A per-IP request budget | Rotate IPs before the budget runs out rather than retrying after it does |
HTTP 403 with a "Just a moment..." page and a cf-ray header | A Cloudflare challenge | A consistent browser fingerprint; a real browser if the page needs JavaScript to pass |
HTTP 403 with an x-datadome header or a datadome cookie | DataDome bot protection | A client whose TLS fingerprint and headers match the browser it claims to be |
_abck and bm_sz cookies, then a 403 or a challenge page | Akamai Bot Manager | Browser-grade TLS, a session that builds up cookies naturally, a slower pace per IP |
| HTTP 200 with a short page, a captcha or "verify you are human" | A soft block that looks like success | Detect it by content, not status code, and retry with a fresh session |
| Different prices or products than you see in your own browser | Content chosen by the visitor's country | A proxy in the country you care about |
A cf-ray header alone only means the site sits behind Cloudflare, as a large share of websites do; it is the challenge page that signals a block.
Lessons from keeping our scrapers running
We maintain over 400 scrapers. These are the lessons that came up again and again, independent of any one site:
A block can look like success
The most expensive failures are the quiet ones. Several protections answer with HTTP 200 and a page that holds no data: a captcha, an interstitial, or an empty shell. A scraper that trusts the status code saves blank rows and moves on. We added checks to our Actors that a page actually contains the expected data, such as a title or a price, before a record is saved or charged. The same check is worth adding to any scraper you write.
Look for the data before fighting the page
Many modern sites render their pages from a JSON API that their own frontend calls, or embed the data in the HTML as a JSON blob. Reading that is usually faster and more stable than parsing visible HTML, and it is sometimes protected less aggressively than the page itself. Open your browser's developer tools, filter network requests by Fetch/XHR, and reload the page before writing any parsing code.
Your headers have to agree with each other
Modern protections compare everything you send: the TLS handshake, HTTP/2 settings, header order and values. A client that claims to be Chrome in its User-Agent but negotiates TLS like a script is easy to spot. Hand-editing one header, for example a custom Accept, can turn a passing request into a blocked one. Use a client built to impersonate a real browser consistently, rather than copying headers by hand.
A real browser is not automatically the answer
Headless browsers pass JavaScript challenges, but they are slower, use far more memory and cost more per page, and some protections detect common headless setups. Use one when a challenge genuinely needs JavaScript, and keep HTTP requests for everything else. Within browsers, details matter: the same page can pass in one browser build and fail in another.
Geography is part of the request
Prices, stock, languages and sometimes access itself depend on where a request comes from. A scraper that works from your laptop can fail or return different data from a server in another country. Choose the proxy country deliberately, and record it with the data so results from different countries never mix.
Which proxies for which job
| Type | Strengths | Watch out for |
|---|---|---|
| Datacenter | Cheap and fast; fine for open APIs and lightly protected sites | Large sites block whole datacenter ranges first |
| Residential | IPs of real home connections in a chosen country; get past most IP reputation checks | Usually billed per GB, so heavy pages get expensive; individual IPs can be slow |
| Mobile | Carrier IPs shared by many real users, rarely blocked outright | The most expensive option; worth it only when the others fail |
Start with the cheapest type that works and move up only when the block tells you to. A proxy does nothing for a fingerprint problem: if every request is blocked from fresh residential IPs, the issue is how you look, not where you come from.
When to stop fighting a site
Getting past a protection once is the easy part; it is the maintenance that costs. Protections change without notice, so a scraper for a heavily protected site needs monitoring, quick fixes and sometimes a different approach every few months. That is the trade-off behind using a maintained scraper: for sites we cover, our Actors handle proxies, sessions and block detection, and we fix them when a site changes. For ideas on what to collect once you have access, see the guides on ecommerce, lead generation and news and social media.
Scraping responsibly
Getting past a block doesn't mean the site has agreed to be scraped. Keep request rates low enough not to affect the site for its users, treat robots.txt as a statement of the owner's expectations, stay away from content behind logins, and be careful with personal data. Our web scraping guide has a short section on the legal side.
Want this data without writing code?
Every scraper on this site runs on Apify with free starter credits.
Related guides
- Web Scraping in 2026: how scrapers work and how to start
- Ecommerce Data Scraping: ecommerce sites are among the most protected targets
- Lead Generation with Web Scraping: directories, scoring and contacts
- News and Social Media Monitoring: sources, filtering and digests