How to Audit Your Site's Crawl Budget (and When It Matters)

Crawl budget is one of the most over-diagnosed problems in SEO. A 300-page brochure site with indexing issues almost never has a crawl budget problem; it has a quality, duplication or internal linking problem. But on a large catalog, a classifieds site or a news publisher, crawl budget is real, and wasting it means new and updated pages take days or weeks to appear in Google.

This guide shows how to tell which situation you are in, and how to run a crawl budget audit when it does matter.

What crawl budget actually is

Google defines crawl budget as the set of URLs Googlebot can and wants to crawl on your site. It is the product of two things:

• Crawl capacity limit. How many parallel connections and requests Googlebot will make without overloading your server. It rises when your server responds quickly and falls when it slows down or returns errors.
• Crawl demand. How much Google wants to crawl your URLs, based on how popular they are, how often they change, and how many URLs it knows about.

If capacity is high but demand is low, Googlebot crawls less than it could. If demand is high but capacity is limited by a slow server, Googlebot crawls less than it wants to. Either way, every request spent on a useless URL is a request not spent on a useful one.

Step 1: Decide whether crawl budget matters for you

Google's own guidance on crawl budget is aimed at two groups: very large sites (on the order of a million or more unique pages) whose content changes moderately often, and medium or large sites (tens of thousands of pages and up) whose content changes very rapidly, such as daily. It also applies to sites where a large share of URLs sits in the "Discovered - currently not indexed" state in Search Console.

If you are in the top row, stop here and read our technical SEO audit checklist instead. If you are anywhere else, continue.

Step 2: Read the Crawl Stats report

In Search Console, open Settings, then Crawl stats. It covers the last 90 days for the property (use a domain property to see all hosts and protocols together).

Read it in this order:

Host status

Check the three host checks: robots.txt fetching, DNS resolution and server connectivity. Any recent failure here directly reduces crawl capacity. A robots.txt that returns server errors is especially damaging, because Google may stop crawling the site until it can fetch the file again.

Totals and average response time

Look at total crawl requests and average response time on the same chart. A sustained rise in response time usually comes with a drop in requests a few days later. That is crawl capacity shrinking to protect your server.

Breakdowns

Click into each breakdown and note the shares:

• By response. A healthy site is overwhelmingly 200 (OK). Large shares of 301, 404 or 5xx mean Googlebot is spending effort on URLs that should not be linked.
• By file type. HTML, JavaScript, CSS, images and JSON. A high share of JSON or "Other" often means API endpoints or tracking calls being crawled.
• By purpose. Discovery (new URLs) versus refresh (known URLs). A site that publishes heavily but shows almost no discovery crawling has a discovery problem: weak internal linking or stale sitemaps.
• By Googlebot type. Smartphone should dominate. Large "Page resource load" shares are normal for JavaScript-heavy sites.

Each breakdown row shows example URLs. Those examples are where the audit really starts.

Step 3: Analyze your server logs

Crawl Stats gives you samples. Server logs give you every request. For a real crawl budget audit you want at least 30 days of access logs from the origin and, if you use a CDN, from the CDN as well, since cached responses may never reach the origin.

Verify that the hits are really Googlebot

Plenty of scrapers set a Googlebot user agent. Verify with a reverse DNS lookup, then a forward lookup on the result:

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The hostname must end in googlebot.com, google.com or googleusercontent.com, and the forward lookup must return the original IP. At scale, match IPs against the ranges Google publishes in its googlebot.json file instead of running lookups per request.

Find where the crawl goes

With a standard combined log format, these one-liners give a quick picture. Run them on verified Googlebot lines only:

# Requests by status code
awk '{print $9}' googlebot.log | sort | uniq -c | sort -rn

# Top crawled paths with query strings stripped
awk '{print $7}' googlebot.log | sed 's/?.*//' | sort | uniq -c | sort -rn | head -50

# Share of requests that carry a query string
awk '$7 ~ /\?/ {q++} END {print q/NR*100 "% with parameters"}' googlebot.log

# Most common parameter names
grep -o '[?&][a-zA-Z_]*=' googlebot.log | sort | uniq -c | sort -rn | head -20

For large sites, load the logs into BigQuery, a log analysis tool or a spreadsheet and group by URL pattern (directory, template, parameter set) rather than by single URL.

Compare against what you want crawled

Join the log data with two other lists:

1. Your canonical, indexable URLs, from your sitemap or a full site crawl.
2. Pages with organic traffic, from Search Console.

This produces the most useful view in the whole audit: valuable URLs that Googlebot rarely visits, and worthless URLs that it visits constantly.

Step 4: Find the common sources of waste

Most crawl waste comes from a short list of causes. Check each against your log patterns.

Faceted navigation deserves special attention

On ecommerce sites, faceted filters are usually the largest single source of waste. A category with eight filters of ten values each can produce millions of crawlable combinations. Decide which filtered views have search demand (for example "red running shoes"), make those clean, indexable pages, and keep the rest out of the crawl. The duplication side of this problem is covered in our guide to duplicate content in ecommerce.

User-agent: *
Disallow: /*?*sort=
Disallow: /*?*sessionid=
Disallow: /*&*&*&
Disallow: /search

The third rule is a blunt instrument: it blocks any URL with three or more ampersands, which on many catalogs means four or more stacked filters. Test patterns like this against real URLs from your logs before deploying them.

Step 5: Fix the causes, in the right order

Work from the changes that reclaim the most crawling with the least risk.

1. Make the server fast and stable

Crawl capacity follows server health. Cache HTML for anonymous users at the edge, fix slow database queries, and make sure your hosting handles crawl spikes. If Crawl Stats shows rising response times, this comes first; no amount of URL cleanup helps if Googlebot is throttling itself.

2. Stop generating useless URLs

Fix the source, not the symptom: remove tracking parameters from internal links, stop linking to every filter combination, return proper 404 status codes for empty pages. URLs that are never linked are rarely crawled for long.

3. Block what remains with robots.txt

For URL spaces you cannot stop generating, robots.txt Disallow is the tool that actually prevents crawling. A noindex tag does not save crawl budget, because Google must still fetch the page to read it. See robots.txt vs noindex vs nofollow for the trade-offs.

Do not use robots.txt to shuffle budget around temporarily. Google advises against toggling Disallow rules to push crawling toward other sections; it does not work that way, and blocked URLs that are linked can still be indexed without content.

4. Strengthen signals for the URLs you want crawled

• Sitemaps. Include only canonical, indexable URLs that return 200. Keep lastmod accurate; Google uses it when it is consistently reliable and ignores priority and changefreq.
• Internal links. Pages linked from high-traffic hubs get crawled more often. Orphaned pages get crawled rarely, if at all. Our guide to internal linking strategy covers how to structure this.
• Consolidate duplicates. Canonical tags and redirects reduce the number of near-identical URLs competing for attention over time.

What no longer exists

Two controls you may remember are gone. Search Console's URL Parameters tool was retired in 2022, and the crawl rate limiter setting was removed in early 2024. If you need to slow Googlebot down urgently, the supported method is to return 503 or 429 responses temporarily; keep it short, since prolonged errors can cause Google to drop URLs from the index.

Re-measure

Crawl behavior changes slowly. After shipping fixes, compare 30-day windows in your logs:

• Share of Googlebot requests to canonical, indexable URLs (should rise).
• Share of 3xx, 4xx and parameter URLs (should fall).
• Time from publishing a page to first Googlebot visit (should shorten).
• "Discovered - currently not indexed" count in the Page indexing report (should trend down).

These are the numbers worth tracking over time. AI Monitoring can watch status codes, response times and sitemap health daily, so a regression that starts eating crawl capacity is caught in days rather than at the next quarterly audit. For deeper Search Console analysis, see our guide to advanced Search Console reports.

Summary

• Crawl budget matters for large or fast-changing sites; most small sites have quality or linking problems instead.
• Start with the Crawl Stats report: host status, response time and the response, file type and purpose breakdowns.
• Use verified Googlebot server logs to compare what is crawled against what you want crawled.
• Fix server speed first, then stop generating junk URLs, then block what remains with robots.txt.
• Remember that noindex does not save crawl budget, and the old parameter and crawl rate tools no longer exist.