Let the AdSense bot in!

Google's just put together a great little help page called How do I block the Googlebot? which could also be titled How do I let in the AdSense bot?. The first part lists the user agents that Google uses for its various crawlers:

• Googlebot (main search index)
• Googlebot-Mobile (mobile search index)
• Googlebot-Image (image search index)
• Mediapartners-Google (AdSense content analysis)
• Adsbot-Google (AdWords landing page analysis)

The rest of the page shows how to use your robots.txt file to allow or disallow the various crawlers to crawl parts of your site.

If you don't let the AdSense crawler see your pages, you won't get optimally-targeted ads. This is because the crawler is what fetches the page so that Google can analyze its content. If it can't fetch the page, ads will probably still show, but the ads will be based solely on the URL of the page (including query parameters), the content of pages linking to that page (if the crawler knows about them) and the content of the other pages on the same site. (Yes, this is all covered in the AdSense patent.) The ads may be relevant in this case, or they may not — it really depends on your site.

With apologies to John and Paul:

When I find my ads are all off target,
Mother Google comes to me,
Speaking words of wisdom:
“Let me in.”
etc. etc.

Sheesh. Enough of that crap.

Pay careful attention to Google's “Allow” extension of the robots.txt standard. The standard only defines a “Disallow” command, but Google's crawlers understand the “Allow” command as well, which gives you more flexibility. Other search engines don't use “Allow”, but if all you want to do is allow the AdSense crawler into pages that you don't want showing up in the main index then that won't matter.

Of course, we'll defer the discussion of why you wouldn't want your AdSense pages showing up in Google's search index to another time…

Most importantly, don't forget that the robots.txt is a voluntary exclusion method. Not every crawler that comes to your site respects it. And don't expose details about specific files you're looking to protect.

Sponsored Link: Hey, sign up for free with ClickBank and make me some money. Please?

Eric Giguere is the contextual advertising expert who wrote Make Easy Money with Google and Uncommon AdSense. You can read this blog by mail if it's more convenient for you, just send a blank email to memwg-blog@aweber.com to subscribe.

Comments