Thanks for the diagnostics! I was expecting it was something like that, but somehow managed to misplace your original report and therefor didn't investigate it further.
I will take a look at it tonight.
//Magnus
On Tue, Jun 24, 2014 at 4:05 PM, Josh Kupershmidt <schmiddy@gmail.com> wrote:
This behavior seems to still be going on, but I think I have a clue. I noticed while experimenting with:
When I fall to 217.196.149.50 and 87.238.57.232, I get the normal robots.txt. When I fall to 174.143.35.230, I get the bad version disallowing all access to the site. BTW, this behavior seems to not be dependent on the user-agent string, contrary to my earlier speculation. Could someone please check out what's going on with robots.txt on 174.143.35.230, as it seems to seriously be screwing with our Google search results. Josh
On Wed, Jun 18, 2014 at 9:26 AM, Josh Kupershmidt <schmiddy@gmail.com> wrote: > I noticed an unusual search result shown as the top result by Google > (search query "POSTGRESQL DROP TRIGGER", first result for me leads to > www.postgresql.org/docs/8.3/static/sql-droptrigger.html ). The title > of the result is somehow "英語 - PostgreSQL", and below that title > reads: "A description for this result is not available because of this > site's robots.txt – learn more." > > Sure enough, when I checked http://www.postgresql.org/robots.txt in > Chrome on OS X, I see: > > User-agent: * > Disallow: / > > though when I check in other browsers (Safari, wget), I see a more > reasonable robots.txt: > > === > User-agent: * > Disallow: /admin/ > Disallow: /account/ > Disallow: /docs/devel/ > Disallow: /list/ > Disallow: /search/ > Disallow: /message-id/raw/ > Disallow: /message-id/flat/ > > Sitemap: http://www.postgresql.org/sitemap.xml > === > > Is it intentional that we're serving up that first robots.txt to > (apparently) Googlebot and Chrome? > > Josh