What is ClaudeBot?
Anthropic’s crawler that fetches pages to inform Claude’s answers. Like other AI crawlers it follows robots.txt, so blocking it is a deliberate distribution decision, not a default.
Related terms
AI Crawler
A bot that fetches web content to train models or ground live answers, distinct from search-index crawlers. Examples include GPTBot, ClaudeBot, PerplexityBot, and Common Crawl’s CCBot.
GPTBot
OpenAI’s crawler, used for training and retrieval. It respects robots.txt and can be disallowed site-wide or per-directory without affecting other crawlers.
AI Bot Blocking
Using robots.txt (or edge rules) to stop AI crawlers from fetching a site. The tradeoff is direct: blocked content cannot be quoted or cited by the assistants doing the blocking.
Answer Engine Optimization (AEO)
Answer Engine Optimization is the practice of structuring pages so answer engines and AI assistants can extract, trust, and quote them: a direct definition up top, scannable steps, quotable facts, and FAQ or Article schema behind them.
Generative Engine Optimization (GEO)
Generative Engine Optimization improves how often a brand or page is cited inside AI-generated answers. It combines classic SEO (crawlability, relevance) with quotability: clear facts, named entities, sources, and machine-readable structure.
AI Overviews
Google’s AI-generated summaries at the top of some results pages. They synthesize multiple sources and link to them, which can reduce clicks to publishers even when a page ranks well organically.