Crawl Budget is the total number of pages a search engine bot (e.g. Googlebot) will crawl on a site within a given period, along with the server resources it will consume.
Crawl budget consists of two components:
- Crawl Capacity Limit: The site server’s capacity to handle bot traffic.
- Crawl Demand: The bot’s desire to crawl this site (high for popular sites that are updated frequently).
Crawl budget becomes especially critical in the following cases:
- Large sites with many pages (e-commerce, forums, news)
- Pages with filtering/sorting parameters
- Sites with frequent content updates
To prevent crawl budget waste, block unnecessary URLs with robots.txt, manage duplicate content with canonical tags, and use noindex tags correctly.
Tip: The “Crawl Stats” report in Search Console shows how Googlebot crawls your site. You can identify crawl errors and unnecessary crawl traffic there.