What is GPTBot?
OpenAI’s crawler, used for training and retrieval. It respects robots.txt and can be disallowed site-wide or per-directory without affecting other crawlers.
Related terms
AI Crawler
A bot that fetches web content to train models or ground live answers, distinct from search-index crawlers. Examples include GPTBot, ClaudeBot, PerplexityBot, and Common Crawl’s CCBot.
OAI-SearchBot
OpenAI’s search-oriented crawler used for live retrieval in ChatGPT answers. It is controlled separately from GPTBot in robots.txt, so training and search-fetch policies can differ.
AI Bot Blocking
Using robots.txt (or edge rules) to stop AI crawlers from fetching a site. The tradeoff is direct: blocked content cannot be quoted or cited by the assistants doing the blocking.
Answer Engine Optimization (AEO)
Answer Engine Optimization is the practice of structuring pages so answer engines and AI assistants can extract, trust, and quote them: a direct definition up top, scannable steps, quotable facts, and FAQ or Article schema behind them.
Generative Engine Optimization (GEO)
Generative Engine Optimization improves how often a brand or page is cited inside AI-generated answers. It combines classic SEO (crawlability, relevance) with quotability: clear facts, named entities, sources, and machine-readable structure.
AI Overviews
Google’s AI-generated summaries at the top of some results pages. They synthesize multiple sources and link to them, which can reduce clicks to publishers even when a page ranks well organically.