I noticed an unusual search result shown as the top result by Google
(search query "POSTGRESQL DROP TRIGGER", first result for me leads to
www.postgresql.org/docs/8.3/static/sql-droptrigger.html ). The title
of the result is somehow "英語 - PostgreSQL", and below that title
reads: "A description for this result is not available because of this
site's robots.txt – learn more."
Sure enough, when I checked http://www.postgresql.org/robots.txt in
Chrome on OS X, I see:
User-agent: *
Disallow: /
though when I check in other browsers (Safari, wget), I see a more
reasonable robots.txt:
===
User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /docs/devel/
Disallow: /list/
Disallow: /search/
Disallow: /message-id/raw/
Disallow: /message-id/flat/
Sitemap: http://www.postgresql.org/sitemap.xml
===
Is it intentional that we're serving up that first robots.txt to
(apparently) Googlebot and Chrome?
Josh