The stories about Crawling & Indexing from this period, ranked by likely impact on a single site.
Ranked by estimated impact on a single site, not by publication time.
Over the years, the Search Central YouTube channel has published hundreds of videos across various series, from Search for Beginners and Ask Googlebot to Search Console Training and Search Off the…
Crawling & Indexing
Decodo has introduced Web Data API, combining scraping, searching, mapping, and crawling under one key for AI agents and retrieval pipelines.
Crawling & Indexing
Google reminds sites to use 503s during PHP patch downtime. Other updates cover robots.txt controls, bracket URLs, and JPEG XL in Google Images and Discover.
Crawling & Indexing
Google's John Mueller was interviewed by David Iwanow, where he shared information on AI-generated content, the future of SEO, technical SEO, the crawl stats report, redirect chains, ecommerce…
Crawling & Indexing Content & Quality
Google is out with a new video in The Non-Canonical Cabinet series, this one is on 404 pages and why they are not scary. I like how they also timed the video with a Halloween theme.
Crawling & Indexing
Google Search has just updated its favicon documentation to now specify that Googlebot must be able to follow redirects from the homepage to the page where the favicon is specified. The document also…
Crawling & Indexing Site Architecture
Google's John Mueller kind of downplayed the importance of the freshness of the Google Search Console page indexing report. He said on Reddit, in response to my coverage of the report's recent…
Crawling & Indexing Measurement
John Mueller's responded to a stalled indexing report and explained why SEOs may need to relax about indexing report freshness.
Crawling & Indexing Measurement
Meta delayed its updated business terms amid complaints to EU and UK regulators. The permission to crawl sites tied to its business users hasn't changed.
Crawling & Indexing
Google's new robots.txt video says the standard covers crawling, not how content gets used, and recommends percent-encoding non-English robots.txt paths.
Crawling & Indexing Content & Quality
Technical SEO is the foundation everything else sits on. The best content in your niche will not rank if search engines cannot crawl it, choose not to index it, cannot render it, or cannot work out…
Ranking & Algorithms Crawling & Indexing Site Architecture
There have been a number of reports recently of sites struggling to be properly indexed following a site migration. I have consulted with a few sites on this issue and have developed a theory. Please…
Crawling & Indexing Site Architecture
Assembled from our own archive of 1,107 stories, which currently runs from 4 October 2023. Ranking is by the impact score described on the front page — an estimate of how much a story is likely to affect one website, not a measure of how much attention it got.
The archive only holds what it has seen. Periods before it started are empty rather than quiet, and it gets more complete every week it runs.
