What is AI Crawler?
A bot that fetches web content to train models or ground live answers, distinct from search-index crawlers. Examples include GPTBot, ClaudeBot, PerplexityBot, and Common Crawl’s CCBot.
Related terms
GPTBot
OpenAI’s crawler, used for training and retrieval. It respects robots.txt and can be disallowed site-wide or per-directory without affecting other crawlers.
ClaudeBot
Anthropic’s crawler that fetches pages to inform Claude’s answers. Like other AI crawlers it follows robots.txt, so blocking it is a deliberate distribution decision, not a default.
PerplexityBot
The crawler behind Perplexity’s answer engine, fetching cited sources for its responses. Being fetchable by it is a precondition for being quoted in Perplexity answers.
Answer Engine Optimization (AEO)
Answer Engine Optimization is the practice of structuring pages so answer engines and AI assistants can extract, trust, and quote them: a direct definition up top, scannable steps, quotable facts, and FAQ or Article schema behind them.
Generative Engine Optimization (GEO)
Generative Engine Optimization improves how often a brand or page is cited inside AI-generated answers. It combines classic SEO (crawlability, relevance) with quotability: clear facts, named entities, sources, and machine-readable structure.
AI Overviews
Google’s AI-generated summaries at the top of some results pages. They synthesize multiple sources and link to them, which can reduce clicks to publishers even when a page ranks well organically.