Yahoo! Crawling to a Halt on GZ files?

July 19, 2007 |

Well, I really hope the news is true but I’ve reason to believe Yahoo is freezing up on a sitemap compression standard that allows us to reduce the file size of large sitemaps. Months ago, Yahoo, Google, and Microsoft (and more recently Ask) announced that they has agreed to the sitemap protocol outlined at sitemaps.org. This tremendous announcement meant that SEOs need only concern themselves with a single sitemap format for structure and discovery.

SEOs working with large websites can use sitemap index files and compressed gzip formats to manage unacceptable file sizes. The protocol permits files of no greater than 50,000 URLs and 10MB; use indexes and multiple sitemaps if you have more URLs, gzip if the files are larger than 10. (interestingly though, even this protocol is inconsistent with sitemaps.org saying, “you may compress your Sitemap files using gzip to stay within 10MB,” while Google states, “no larger than 10MB when uncompressed.”)

At this point I’m dealing in speculation in hopes of soliciting your own experience. Yahoo doesn’t seem to like gz files and while it may be due to the inconsistency in the file size spec, a bug in Yahoo’s crawlers, or their simple disfavor of the sites I’m working with, log files indicate problems. Is anyone else using gzip on their sitemaps? What’s going on!?

Popularity: 46% [?]

If you enjoyed this post, make sure you subscribe to my RSS feed!

Tags: yahoo, seo, sitemaps