How to scrape Google in 2026

What “scraping Google” means

Scraping Google means automatically collecting the results page of a search (the SERP) to extract its data: organic results, ads, local pack, related questions, AI Overviews, and their position. The uses are well known: SEO rank tracking, competitor monitoring, market research, keyword clustering by SERP similarity, and measuring visibility in AI-generated answers.

Three ways to do it

1. By hand or with a browser extension

Fine for a few dozen keywords. Beyond that it is too slow, and your own history, location and Google account skew the results.

2. An in-house scraper

A script that queries Google itself. Free on paper, but in practice you need:

• proxies, ideally residential, and their rotation, because Google quickly blocks addresses that query it in bulk;
• captcha handling;
• a headless browser to render the page, or part of the content is missing (see below);
• a parser, to maintain every time Google changes its layout;
• handling of country, language, city and device.

The real cost is proxies, servers and above all maintenance time.

3. A SERP API

A service that does all of this for you: send a keyword and its parameters, get the structured SERP back in JSON. Most SEO tools go this way, preferring to pay per request rather than run a scraping infrastructure. To pick a provider, see our comparisons with SerpApi and DataForSEO.

What a SERP contains in 2026

A results page is no longer a list of ten blue links. Depending on the query, Google adds:

• organic results;
• Google Ads at the top and bottom of the page, and Shopping ads;
• the local pack, with each business's rating, review count and price range;
• “People also ask”, the featured snippet and the Knowledge Graph;
• top stories, videos and images;
• the AI Overview, with the brands cited in its text, its citations and its source panel.

To measure what is actually visible, each block has to be extracted separately, with the position of each of its elements. The Semscraper API returns every block under its own type: organic, paid_top, local_pack, people_also_ask, ai_overview_citation…

The four pitfalls of 2026

The num=100 parameter no longer works

Since September 2025, Google no longer shows 100 results on a single page. To get a top 100 you have to walk through the pages one by one. We cover the method and its cost in Getting Google's top 100 without num=100.

AI Overviews arrive after the page loads

In our study of one million keywords, 48% of AI Overviews load asynchronously: missing from the initial HTML, they only appear once the page is displayed. A scraper that doesn't render the page misses them. Details in our study of AI Overviews in France.

The SERP depends on the city

A local search, “plumber” or “restaurant”, doesn't return the same results in two different cities. Without geolocation you measure a SERP nobody sees. See geolocated SERPs.

Desktop and mobile diverge

One keyword in eight behaves differently on the two devices for AI Overviews, and block order differs. Collect the device that matters for your use case, or both.

Example with the Semscraper API

A POST request creates the collection, with up to 1,000 keywords per call:

curl -X POST 'https://api.semscraper.com/v1/serp' \
  -H 'Authorization: Bearer API_KEY' \
  -H 'Content-Type: application/json' \
  -d '[{"search_engine": "google_search", "keyword": "restaurant", "device": "desktop", "location": "fr", "language": "fr", "depth": 1, "geolocation": "Nice, Alpes-Maritimes, France"}]'

Results are then fetched by ID, in JSON or HTML, or delivered straight to your callback URL. Every result carries its rank within its block, its rank on the page, its Google page and its pixel position. The full list of blocks and parameters is on the Google Search API page.

Search operators work

Google's operators work as-is inside the keyword: site:, inurl:, intitle:, filetype:, quotes, or excluding a term with a minus sign. Handy to list a site's indexed pages or watch a brand on a given domain. A query with an operator costs the same as any other. One constraint: no comma in a keyword, the API uses it as a separator.

Complete Python example

This script sends three keywords, waits for their results, then prints for each one whether there is an AI Overview and the top three organic results with their pixel position:

import time
import requests

API = "https://api.semscraper.com/v1/serp"
HEADERS = {"Authorization": "Bearer API_KEY"}

keywords = ["restaurant austin", "plumber denver", "divorce lawyer chicago"]

# 1. create the collections, up to 1,000 keywords per call
items = [{"search_engine": "google_search", "keyword": k, "device": "desktop",
          "location": "en", "language": "en", "depth": 1} for k in keywords]
created = requests.post(API, json=items, headers=HEADERS).json()
ids = [row["id"] for row in created["data"]]

# 2. fetch the results (500 IDs at most per call), 10 minutes at most
done, attempts = {}, 0
while len(done) != len(ids) and attempts < 60:
    attempts += 1
    time.sleep(10)
    res = requests.get(API, params={"ids": ",".join(ids), "output": "json"}, headers=HEADERS).json()
    for row in res["data"]:
        if row["status"] == "done":
            done[row["id"]] = row

# 3. read each SERP: AI Overview, then the top 3 organic results
for row in done.values():
    aio = row["serp_info"].get("ai_overview")
    print(row["keyword"], "| AI Overview:", aio["type"] if aio else "-")
    for block in row["results"]:
        if block["type"] == "organic":
            for item in block["items"][:3]:
                print("   ", item["rank_type"], item["pixel"], "px", item["url"])

For large volumes, replace the polling loop with a callback URL (callback_url): the API sends you each result as soon as it is ready.

Volumes and turnaround

A one-page SERP arrives in a median of about thirty seconds, and a top 100 in about a minute. Each call can carry up to 1,000 keywords. The robots serve accounts in turn: another customer's large batch doesn't hold up yours, and yours flows through your own queue.

Choosing a SERP API: the checklist

• Real pagination, with each result's page, for a reliable top 100.
• Rendering in a browser, to capture AI Overviews that load late.
• City-level geolocation, not just country.
• Pixel position, to measure a result's real visibility.
• Billing for pages found only, with no forced subscription.
• The page HTML on top of the JSON, to check a surprising result.
• A fair queue between customers.

What it costs

With Semscraper, the first page costs €0.30 per 1,000 keywords and each extra page €0.20, so €2.10 per 1,000 keywords over 10 pages. Only pages found are billed, with no subscription. Details on the pricing page.

What about legality?

Results pages are public, but collecting them automatically is still governed by Google's terms of service and, if you process personal data, by data protection law such as the GDPR. This guide is not legal advice: have your use case reviewed if in doubt.