What is llms.txt?
An experimental convention (a Markdown file at /llms.txt) that summarizes a site for language models. Adoption and crawler support are still thin, so it sits below core crawlability and content quality in priority.
Related terms
AI Crawler
A bot that fetches web content to train models or ground live answers, distinct from search-index crawlers. Examples include GPTBot, ClaudeBot, PerplexityBot, and Common Crawl’s CCBot.
Robots.txt
A plain-text file at the domain root telling compliant crawlers which paths they may fetch. It controls crawling, not indexing: blocked URLs can still be indexed by URL if linked.
Machine-Readable Content
Content structured so parsers extract it reliably: semantic HTML, one H1, real lists and tables, descriptive link text, alt text, and schema that mirrors visible content.
Answer Engine Optimization (AEO)
Answer Engine Optimization is the practice of structuring pages so answer engines and AI assistants can extract, trust, and quote them: a direct definition up top, scannable steps, quotable facts, and FAQ or Article schema behind them.
Generative Engine Optimization (GEO)
Generative Engine Optimization improves how often a brand or page is cited inside AI-generated answers. It combines classic SEO (crawlability, relevance) with quotability: clear facts, named entities, sources, and machine-readable structure.
AI Overviews
Google’s AI-generated summaries at the top of some results pages. They synthesize multiple sources and link to them, which can reduce clicks to publishers even when a page ranks well organically.