Crawler (spider/bot)
The automated visitor that follows links and fetches pages for a search engine. Googlebot is one; a human with Chrome is not.
See it
What it is
A crawler is an automated program that requests URLs and reads what comes back. Googlebot is the famous one. It fetches a page, extracts links, queues URLs it has not visited yet, and repeats. Search engines also seed crawlers with XML sitemaps and URLs they already know from elsewhere on the web.
Reach for this term when you are reading server logs, deciding which bots to permit, or asking how a new page gets discovered. A crawler usually identifies itself in the HTTP User-Agent header, and Googlebot's ends with 'Googlebot/2.1; +http://www.google.com/bot.html'. Any script can copy that string, which is why verification matters more than the label. The response it sees matters too: a 200 with useful HTML, a redirect, a 404, and a 403 all send different signals.
The gotcha is the handoff. A crawler fetches; an indexer decides what to store; ranking systems decide what to show. Seeing Googlebot in a log proves that a request happened, not that the page was indexed or will rank.
Ask AI for it
Audit this site's crawler traffic from its HTTP access logs. Separate Googlebot and Bingbot requests by User-Agent, then verify claimed Googlebot addresses with Google's forward and reverse DNS technique. Report requested URLs, HTTP status codes, redirect chains, crawl frequency, and repeated hits to parameter or faceted-navigation URLs. Recommend robots.txt rules only for crawl waste, never as a way to remove an indexed URL.