Web Scraping in 2026: How It Works and How to Start
Web scraping means collecting information from websites automatically and saving it in a structured form, such as a spreadsheet or a database. Instead of copying 60 agency profiles from a directory by hand, a scraper reads the pages and returns one row per agency, with the same columns every time.
This guide explains what a scraper does, the ways to start depending on how much code you want to write, what real runs cost, and where things usually go wrong. The topic guides then go deeper on ecommerce, lead generation and news and social media.
What a scraper does
Every scraper, from a ten-line script to a production system, does three things:
- Request. It loads a page or calls the API that the site's own frontend uses, the same way a browser would.
- Extract. It finds the values it needs, like a price, a rating or a company name, and turns them into named fields.
- Store. It saves one record per item and moves on to the next page until it has what it needs.
Here is what that produces for one company from a Clutch directory page, taken from our example run on October 7, 2026:
| Field | Value |
|---|---|
| Company | Atomic Object |
| Location | Chicago, IL |
| Team size | 50 - 249 |
| Hourly rate | $150 - $199 / hr |
| Minimum project | $25,000+ |
| Rating and reviews | 4.9 from 52 reviews |
Sixty of these rows, one per agency on two directory pages, is something you can sort, filter and import into a CRM. The same page as HTML is not.
The hard parts are rarely the extraction. They are getting the page at all when a site blocks automated traffic, which our guide on scraping without getting blocked covers, and keeping the scraper working when the site changes its layout.
Ways to start
| Option | What you do | Good for |
|---|---|---|
| Ready-made scraper on Apify | Fill in a form (a search, a URL, a limit), run it and export CSV, JSON or Excel | Most people: no code, pay per result, maintained by someone else |
| Your own code | Write a scraper with a library such as Crawlee or Playwright | Sites nobody covers, or very custom logic |
| A complete pipeline | Clone a project that runs a scraper on a schedule, stores the history in Postgres and adds an AI step | Recurring monitoring: price history, lead scoring, mention tracking |
A ready-made scraper on Apify is called an Actor. All the scrapers we link to are our own Actors, and each of the 110+ site guides on our blog walks through one run with real input and output. Writing your own makes sense when no Actor covers the site, but budget for maintenance: sites change layouts and protections, and a scraper nobody watches tends to fail quietly.
What it costs
Most Actors charge per result, so the cost follows the amount of data rather than time or servers. These are real runs from our own projects, priced as of October 7, 2026:
| Run | Results | Cost |
|---|---|---|
| Agencies from two Clutch directory pages | 60 companies | $0.09 |
| Open roles at five companies on LinkedIn, with full job details | 60 jobs | $0.23 |
| One Mercado Libre search | 50 listings | $0.10 |
| Mentions of one brand on Threads, Reddit and Google News | 89 posts and articles | $0.21 |
| A daily price check of seven AliExpress products | 7 products a day | About $1 a month |
The Apify free plan includes $5 of monthly usage, which covers small experiments like these many times over. Each site guide states the current price per result for its Actor.
Five complete projects
If you want to see what a scraper looks like inside a real workflow, these open-source projects each run our Actors on a schedule, keep the history in Postgres and add an AI step with Claude:
| Project | What it does |
|---|---|
| AliExpress price tracker | Daily prices and stock for a watchlist, with a buy now or wait call per product |
| Mercado Libre price tracker | Market prices for a search, grouped into comparable models |
| Clutch lead generation pipeline | Agencies scored against your ideal customer, then decision makers at the best fits |
| LinkedIn jobs AI matcher | Jobs at the companies you follow, ranked against your resume |
| Brand mention monitor | Mentions from three sources, filtered and summarized into a digest |
Data that looks right but isn't
A scraper that runs without errors can still hand you misleading data. These are the traps we see most:
- Displayed numbers that mean something else. A crossed-out list price is not what the product sold for: in our AliExpress runs, every product showed a 34% to 84% discount on every day we checked. Use the product's own price history as the baseline.
- Mixed items in one search. A search for "iphone 15" returns the 15, Plus, Pro and Pro Max, new and refurbished. Group like with like before averaging anything.
- Matches that aren't about your topic. A keyword search returns everything containing the word. In our brand monitoring run, 23 of 74 matches were job ads, skill lists or posts that never mentioned the brand.
- Location-dependent results. Prices, stock and even which products exist can change with the country of the visitor. Keep one country and currency per dataset.
- Duplicates across pages. Sponsored items and shifting rankings can show the same item twice. Deduplicate by the site's own ID.
Legal and responsible use
The rules depend on what you collect, where you and the people in the data are, and what you do with it, so treat this as orientation rather than legal advice:
- Public pages are the safest starting point. Avoid scraping behind a login, and read the site's terms of service before relying on its data commercially.
- Personal data has its own rules. Names, emails and profiles of people fall under laws such as the GDPR in the EU and UK. The UK ICO's guidance on legitimate interests explains what processing such data for business purposes involves.
- Don't overload the site. Keep request rates reasonable, treat
robots.txtas a signal of what the site owner expects, and don't republish copyrighted content such as full articles.
Want this data without writing code?
Every scraper on this site runs on Apify with free starter credits.
Related guides
- How to Scrape Without Getting Blocked: recognize which protection you hit and what helps
- Ecommerce Data Scraping: prices, suppliers and resale values
- Lead Generation with Web Scraping: directories, scoring and contacts
- News and Social Media Monitoring: sources, filtering and digests