[Webmaster Tools] http://www.example.com/: Googlebot can’t access your site

by · Published · Updated

I am posting this unsolved mystery – as an ongoing problem – instead of posting to a forum. Comment if you like. I’l update when I find the solution.

I get the typical message from Google [Webmaster Tools] http://www.example.com/: Googlebot can’t access your site.

fetch-as-google-bot

I run LAMP servers with Netfilter (iptables) with PSAD and Fail2ban.

I check iptables for any trace of Google IPs having been caught (banned). Nothing.

I watch the logs to find this (when I manually tell Google to try fetching again):

==> /var/log/apache2/error.log <== [Mon Feb 23 08:52:02 2015] [debug] mod_deflate.c(700): [client 66.249.79.129] Zlib: Compressed 5855 to 1326 : URL /index.php
==> /var/log/apache2/other_vhosts_access.log <== [23/Feb/2015:08:52:02 +0000] example.com:80 66.249.79.129 - - "GET /robots.txt HTTP/1.1" 200 2110 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

There is no error and nothing indicating that Googlebot was denied access.

I flush iptables. And tell Google to try again and get this

==> /var/log/apache2/error.log <== [Mon Feb 23 09:03:26 2015] [debug] mod_deflate.c(700): [client 66.249.79.137] Zlib: Compressed 5855 to 1326 : URL /index.php
==> /var/log/apache2/other_vhosts_access.log <== [23/Feb/2015:09:03:25 +0000] example.com:80 66.249.79.137 - - "GET /robots.txt HTTP/1.1" 200 2110 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
==> /var/log/apache2/error.log <== [Mon Feb 23 09:03:26 2015] [debug] mod_deflate.c(700): [client 66.249.79.129] Zlib: Compressed 47767 to 10101 : URL /index.php
==> /var/log/apache2/other_vhosts_access.log <==
[23/Feb/2015:09:03:26 +0000] example.com:80 66.249.79.129 - - "GET / HTTP/1.1" 200 10839 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Tags:

You may also like...