I am not really sure how long this feature has been available, but I think it's hot !


First time in search engine history you have a chance to get peek into the web spider / crawler's mind with feedback about urls that could not be crawled and why !

Steps to use this feature:


# Submit a sitemap to Google at https://www.google.com/webmasters/sitemaps/

1. Upload a special and empty file to the same directory as your sitemap file, name some cryptic like GOOGLExyxyl329.html so that Google can identify that you – owner of the sitemap account really have control over the site you intend to get information about

1. Click the verify button to lookup that file

1. Now the "verify" button changes to "stats" and you can get very nice messages like

Site: http://www.cemper.com/

URLs found during our regular crawl process

We have been crawling your site as part of our regular crawl process. This includes following links from your pages and the pages of other sites. Below we have listed URLs that we were unable to reach during this crawl process, with links that explain why we could not reach them.

Followed by list of files and urls (sometimes just links from external that have typos)

Major benefit is of course to identify if any external sites link to your aged pages that don't exist anymore… now go setup a redirect and save some link juice again…