Google, Yahoo, Microsoft Team Up for Crawling

Liz Gannes | Wednesday, November 15, 2006 | 10:46 PM PT | 2 comments

Google has brought Yahoo and Microsoft on board with a standard format, called Sitemaps, for web masters to submit their pages for spidering and indexing by search engines. In order to get quicker access to dynamic content, the companies will encourage webmasters and content management systems to place a Sitemap XML file on their servers. Publishers will only have to generate one of these feeds to get their most recent content considered for inclusion in search results across all the participating engines.

Sitemaps is going to help especially with surfacing updates from complex URLs and frequently changing pages at e-commerce sites and user-generated content networks, said product managers Vanessa Fox of Google and Tim Mayer of Yahoo. Mayer (whose full title is senior director of product management at Yahoo Search) said the Sitemaps format is currently “very simplistic,” but it could be expanded to include meta information about URLs and geo-location data.

Google had first released Sitemaps over a year ago, while Yahoo and then Microsoft signed on recently. The format is now open to other search engines as well. We gotta say, it’s pretty funny to get an email from Google to set up a partnership announcement phone call (with no more details than just that), and then dial in to hear a recorded voice say “Welcome to conferencing at Yahoo!”

Danny Sullivan writes at Search Engine Watch,

Overall, I’m thrilled. It took nearly a decade for the search engines to go from unifying around standards for blocking spidering and making page description to agreeing on the nofollow attribute for links in January 2005. A wait of nearly two years for the next unified move is a long time, but far less than 10 and progress that’s very welcomed. I applaud the three search engines for all coming together and look forward to more to come.

Comments (2)

Linkbacks (1)

  • Sitebases, the next protocol after Sitemaps…

    It can save the time and press for the search engine, also for the websites.
    It can bring new search engine that named Search Engin 2.0.
    Using Sitebase protocol, will save 95% bandwidth above. It is anthother sample for long tail thery.
    In this protoco…

     

Subscribe to comments feed

Leave a Reply

Editorial Masthead

Carolyn Pritchard
Managing Editor
Celeste LeCompte
Special Projects Editor
Desiree DeNunzio
Copyeditor
Om Malik
Senior Writer
Stacey Higginbotham
Staff Writer
Jennifer Martinez
Staff Writer
Wagner James Au
Contributing Editor
Liz Gannes
Staff Writer
Chris Albrecht
Staff Writer
Katie Fehrenbacher
Staff Writer
Josie Garthwaite
Staff Writer
Close
E-mail It