Googlebot and Mediabot – Can you get in trouble?

May 16, 2006

On the PubCon Publishers Search and Marketing Conferences in Boston, Google’s Matt Cutts revealed interesting information about Mediabot and how it could influence your Google rankings.

What are Googlebot and Mediabot?

Search engines use software programs (so-called bots or spiders) to index web pages. These bots follow all links they can find on the Internet to index web pages for their search engines.

The name of Google’s bot is Googlebot. If you see an entry with the name Googlebot in your web site statistics then Googlebot has visited your web site.

In addition to Googlebot, Google also operates another bot with the name Mediabot. Mediabot is the bot that is used by Google to index web sites that display AdSense ads. Google indexes these pages to find out which ads Google should display on these pages.

What’s the problem with Mediabot?

Until recently, Google has denied that displaying AdSense ads on your web pages has any effect on your Google rankings. However, some webmasters noticed that web pages that have only been visited by Mediabot and not by Googlebot showed up in Google’s search results.

Google’s Matt Cutts confirmed at the Boston PubCon 2006 that the MediaBot pages can also be used for Google’s main index:

“Pages with AdSense will not be indexed more frequently. It’s literally just a crawl cache, so if e.g. our news crawl fetched a page and then Googlebot wanted the same page, we’d retrieve the page from the crawl cache. But there’s no boost at all in rankings if you’re in AdSense or Google News. You don’t get any more pages crawled either.”

How can this get you in trouble?

If you serve different pages to Googlebot and Mediabot then Google might ban your web site because you use cloaking on your web site. Cloaking means that you’re showing different pages to users than to search engines.

As Mediabot analyzes the web pages you serve for your visitors it’s easy for Google to find out if you present extra pages for Googlebot. If you do something special for Googlebot then Google knows that you’re cloaking.


Comments on this entry are closed.

Previous post:

Next post: