What is Log File Analysis?
Reading server logs to see which URLs bots actually fetch, how often, and with what status. Logs reveal crawl waste and orphaned discovery that crawler simulations can only guess at.
Related terms
Crawling
How search-engine bots discover pages by following links, sitemaps, and directives. If a URL is never crawled, it cannot be indexed no matter how good its content is.
Crawl Budget
The number of pages a search engine will crawl on a site in a given period, driven by server capacity and demand signals. Large or slow sites must spend it on indexable, valuable URLs.
Crawl Stats Report
The Search Console report showing crawl volume, response times, and status breakdowns over time. Sudden shifts after releases flag regressions before rankings move.
Technical SEO
The crawling, indexing, rendering, and performance work that lets search engines reach, understand, and serve your pages. It covers directives, architecture, speed, and structured data rather than words on the page.
Indexing
Storing and organizing crawled pages so they can be retrieved for queries. Indexing is selective: crawled does not mean indexed, and indexed does not mean ranking.
Robots.txt
A plain-text file at the domain root telling compliant crawlers which paths they may fetch. It controls crawling, not indexing: blocked URLs can still be indexed by URL if linked.