Crawl budget
How much fetching a search bot will spend on your site. Usually a large-site concern, not a quota small sites need to chase.
See it
What it is
Crawl budget is the amount of fetching a search engine is willing and able to do on one site over time. It is not a fixed daily credit. For Google, the practical ceiling comes from crawl capacity, meaning what the server can handle safely, and crawl demand, meaning which URLs seem important or changed enough to revisit.
Reach for it on very large sites, rapidly changing inventory, or sites where logs show bots spending most requests on faceted filters, calendar traps, duplicate parameters, or long redirect chains. Help the crawler by keeping sitemaps current, linking important pages clearly, returning fast and accurate status codes, and closing infinite URL spaces. Small sites with a few thousand stable URLs usually have a discovery or quality problem, not a crawl-budget problem.
Gotcha: robots.txt can stop a fetch, but it does not remove a URL from search. Noindex needs the page to remain crawlable so the bot can read the directive, and canonical is a duplicate hint rather than a command to stop fetching. Diagnose from server logs and Google Search Console's Crawl Stats report before cutting off paths.
Ask AI for it
Audit this site for crawl-budget waste using raw server logs and the Google Search Console Crawl Stats report. Separate verified Googlebot requests by URL pattern, status code, response time, file type, and crawl frequency; then rank the patterns by requests spent on duplicate filters, parameters, internal search, calendar URLs, redirects, 404s, and 5xx responses. For each waste pattern, prescribe one concrete fix using RFC 9309 robots.txt rules, direct 301 redirects, 404 or 410 responses, canonical tags, or internal-link removal as appropriate. Keep every noindex URL crawlable, preserve valuable pages, and produce a before-and-after crawl map plus an XML sitemap containing only canonical 200-status URLs.