The stories about Crawling & Indexing from this period, ranked by likely impact on a single site.
Ranked by estimated impact on a single site, not by publication time.
Decodo has introduced Web Data API, combining scraping, searching, mapping, and crawling under one key for AI agents and retrieval pipelines.
Crawling & Indexing
Google's John Mueller was interviewed by David Iwanow, where he shared information on AI-generated content, the future of SEO, technical SEO, the crawl stats report, redirect chains, ecommerce…
Crawling & Indexing Content & Quality
Google is out with a new video in The Non-Canonical Cabinet series, this one is on 404 pages and why they are not scary. I like how they also timed the video with a Halloween theme.
Crawling & Indexing
Google Search has just updated its favicon documentation to now specify that Googlebot must be able to follow redirects from the homepage to the page where the favicon is specified. The document also…
Crawling & Indexing Site Architecture
Google's John Mueller kind of downplayed the importance of the freshness of the Google Search Console page indexing report. He said on Reddit, in response to my coverage of the report's recent…
Crawling & Indexing Measurement
John Mueller's responded to a stalled indexing report and explained why SEOs may need to relax about indexing report freshness.
Crawling & Indexing Measurement
Meta delayed its updated business terms amid complaints to EU and UK regulators. The permission to crawl sites tied to its business users hasn't changed.
Crawling & Indexing
New spam guidance says to noindex or block crawling on pages made for sharing. Google's docs say a URL blocked in robots.txt can be indexed.
Crawling & Indexing
Assembled from our own archive of 1,106 stories, which currently runs from 4 October 2023. Ranking is by the impact score described on the front page — an estimate of how much a story is likely to affect one website, not a measure of how much attention it got.
The archive only holds what it has seen. Periods before it started are empty rather than quiet, and it gets more complete every week it runs.
