News Update Place

Yet another Google spider is going to be showing up — AdsBot-Google will be used to monitor AdWords landing page quality.

While we strongly recommend against restricting our system’s automatic review of your landing page, you can edit your site’s robots.txt file to avoid a review. The file must explicitly exclude your page from our system visits as follows:

To prevent AdsBot-Google from accessing your site, add the following to your robots.txt file:
User-agent: AdsBot-Google
Disallow: /

To prevent AdsBot-Google from accessing parts of your site, add the following to your robots.txt file:
User-agent: AdsBot-Google
Disallow: /exclude/

http://adwords.google.com/support/bin/answer.py?answer=38197

Depending on any current robots.txt restrictions, we also might want to be sure we aren’t accidentally excluding the AdsBot. What isn’t clear to me is whether AdsBot will also be participating in the big cache sharing free-for-all along with all the other googlebots. Also, I hope the user agent includes the string “googlebot” so various stats packages automatically catch it as a Google spider.