How to Scrape Without Getting Blocked: Recognize the Block, Then Fix It

Most blocking problems are fixed in the wrong order. A scraper fails, and the first reaction is to add proxies, switch to a headless browser or slow everything down, often all at once. It is faster to first work out which kind of protection stopped you, because each kind responds to a different fix, and some of them don't respond to proxies at all.

This guide shows how to read a block, what each kind usually needs, which proxies fit which job, and the lessons we learned keeping our own scrapers running. If you'd rather not deal with any of this, the scrapers in our catalogue handle it for the sites they cover.

Read the block first

Look at the status code, the response headers and the first lines of the body before changing anything. These are the patterns we run into most:

What you seeUsually meansWhat tends to help
HTTP 429, sometimes with a Retry-After headerA plain rate limitSlow down, respect Retry-After, spread requests over more IPs
Works for a few dozen requests, then every request fails from that IPA per-IP request budgetRotate IPs before the budget runs out rather than retrying after it does
HTTP 403 with a "Just a moment..." page and a cf-ray headerA Cloudflare challengeA consistent browser fingerprint; a real browser if the page needs JavaScript to pass
HTTP 403 with an x-datadome header or a datadome cookieDataDome bot protectionA client whose TLS fingerprint and headers match the browser it claims to be
_abck and bm_sz cookies, then a 403 or a challenge pageAkamai Bot ManagerBrowser-grade TLS, a session that builds up cookies naturally, a slower pace per IP
HTTP 200 with a short page, a captcha or "verify you are human"A soft block that looks like successDetect it by content, not status code, and retry with a fresh session
Different prices or products than you see in your own browserContent chosen by the visitor's countryA proxy in the country you care about

A cf-ray header alone only means the site sits behind Cloudflare, as a large share of websites do; it is the challenge page that signals a block.

Lessons from keeping our scrapers running

We maintain over 400 scrapers. These are the lessons that came up again and again, independent of any one site:

A block can look like success

The most expensive failures are the quiet ones. Several protections answer with HTTP 200 and a page that holds no data: a captcha, an interstitial, or an empty shell. A scraper that trusts the status code saves blank rows and moves on. We added checks to our Actors that a page actually contains the expected data, such as a title or a price, before a record is saved or charged. The same check is worth adding to any scraper you write.

Look for the data before fighting the page

Many modern sites render their pages from a JSON API that their own frontend calls, or embed the data in the HTML as a JSON blob. Reading that is usually faster and more stable than parsing visible HTML, and it is sometimes protected less aggressively than the page itself. Open your browser's developer tools, filter network requests by Fetch/XHR, and reload the page before writing any parsing code.

Your headers have to agree with each other

Modern protections compare everything you send: the TLS handshake, HTTP/2 settings, header order and values. A client that claims to be Chrome in its User-Agent but negotiates TLS like a script is easy to spot. Hand-editing one header, for example a custom Accept, can turn a passing request into a blocked one. Use a client built to impersonate a real browser consistently, rather than copying headers by hand.

A real browser is not automatically the answer

Headless browsers pass JavaScript challenges, but they are slower, use far more memory and cost more per page, and some protections detect common headless setups. Use one when a challenge genuinely needs JavaScript, and keep HTTP requests for everything else. Within browsers, details matter: the same page can pass in one browser build and fail in another.

Geography is part of the request

Prices, stock, languages and sometimes access itself depend on where a request comes from. A scraper that works from your laptop can fail or return different data from a server in another country. Choose the proxy country deliberately, and record it with the data so results from different countries never mix.

Which proxies for which job

TypeStrengthsWatch out for
DatacenterCheap and fast; fine for open APIs and lightly protected sitesLarge sites block whole datacenter ranges first
ResidentialIPs of real home connections in a chosen country; get past most IP reputation checksUsually billed per GB, so heavy pages get expensive; individual IPs can be slow
MobileCarrier IPs shared by many real users, rarely blocked outrightThe most expensive option; worth it only when the others fail

Start with the cheapest type that works and move up only when the block tells you to. A proxy does nothing for a fingerprint problem: if every request is blocked from fresh residential IPs, the issue is how you look, not where you come from.

When to stop fighting a site

Getting past a protection once is the easy part; it is the maintenance that costs. Protections change without notice, so a scraper for a heavily protected site needs monitoring, quick fixes and sometimes a different approach every few months. That is the trade-off behind using a maintained scraper: for sites we cover, our Actors handle proxies, sessions and block detection, and we fix them when a site changes. For ideas on what to collect once you have access, see the guides on ecommerce, lead generation and news and social media.

Scraping responsibly

Getting past a block doesn't mean the site has agreed to be scraped. Keep request rates low enough not to affect the site for its users, treat robots.txt as a statement of the owner's expectations, stay away from content behind logins, and be careful with personal data. Our web scraping guide has a short section on the legal side.

Want this data without writing code?

Every scraper on this site runs on Apify with free starter credits.

Related guides