- Three options: by hand, an in-house scraper (proxies, captchas, a parser to maintain) or a SERP API.
- Four pitfalls in 2026: num=100 is gone, asynchronous AI Overviews, SERPs that vary by city and by device.
- With Semscraper: €0.30 per 1,000 keywords for page 1, €2.10 for a top 100, no subscription.
What “scraping Google” means
Scraping Google means automatically collecting the results page of a search (the SERP) to extract its data: organic results, ads, local pack, related questions, AI Overviews, and their position. The uses are well known: SEO rank tracking, competitor monitoring, market research, and measuring visibility in AI-generated answers.
Three ways to do it
1. By hand or with a browser extension
Fine for a few dozen keywords. Beyond that it is too slow, and your own history, location and Google account skew the results.
2. An in-house scraper
A script that queries Google itself. Free on paper, but in practice you need:
- proxies, ideally residential, and their rotation, because Google quickly blocks addresses that query it in bulk;
- captcha handling;
- a headless browser to render the page, or part of the content is missing (see below);
- a parser, to maintain every time Google changes its layout;
- handling of country, language, city and device.
The real cost is proxies, servers and above all maintenance time.
3. A SERP API
A service that does all of this for you: send a keyword and its parameters, get the structured SERP back in JSON. Most SEO tools go this way, preferring to pay per request rather than run a scraping infrastructure.
The four pitfalls of 2026
The num=100 parameter no longer works
Since September 2025, Google no longer shows 100 results on a single page. To get a top 100 you have to walk through the pages one by one. We cover the method and its cost in Getting Google's top 100 without num=100.
AI Overviews arrive after the page loads
In our study of one million keywords, 48% of AI Overviews load asynchronously: missing from the initial HTML, they only appear once the page is displayed. A scraper that doesn't render the page misses them. Details in our study of AI Overviews in France.
The SERP depends on the city
A local search, “plumber” or “restaurant”, doesn't return the same results in two different cities. Without geolocation you measure a SERP nobody sees. See geolocated SERPs.
Desktop and mobile diverge
One keyword in eight behaves differently on the two devices for AI Overviews, and block order differs. Collect the device that matters for your use case, or both.
Example with the Semscraper API
A POST request creates the collection, with several keywords per call if needed:
curl -X POST 'https://api.semscraper.com/v1/serp' \
-H 'Authorization: Bearer API_KEY' \
-H 'Content-Type: application/json' \
-d '[{"search_engine": "google_search", "keyword": "restaurant", "device": "desktop", "location": "fr", "language": "fr", "depth": 1, "geolocation": "Nice, Alpes-Maritimes, France"}]'Results are then fetched by ID, in JSON or HTML, or delivered straight to your callback URL. Every result carries its rank within its block, its rank on the page, its Google page and its pixel position. The full list of blocks and parameters is on the Google Search API page.
What it costs
With Semscraper, the first page costs €0.30 per 1,000 keywords and each extra page €0.20, so €2.10 per 1,000 keywords over 10 pages. Only pages found are billed, with no subscription. Details on the pricing page.
What about legality?
Results pages are public, but collecting them automatically is still governed by Google's terms of service and, if you process personal data, by data protection law such as the GDPR. This guide is not legal advice: have your use case reviewed if in doubt.
