Hi,
This got brought up again on in a twitter discussion, see
https://twitter.com/AndresFreundTec/status/1403418002951794688
On 2019-08-29 07:50:13 -0700, Andres Freund wrote:
> > > Why is that /list/ exclusion there in the first place?
>
> > Because there are basically infinite number of pages in that space, due to
> > the fact that you can pick an arbitrary point in time to view from.
>
> You mean because of the per-day links, that aren't really per-day? I
> think the number of links due to that would still be manageable traffic
> wise? Or are they that expensive to compute? Perhaps we could make the
> "jump to day" links smarter in some way? Perhaps by not including
> content for the following days in the per-day pages?
I still don't understand why all of /list/ is in robots.txt. I
understand why we don't necessarily want to index /list/.../since/...,
but prohibiting all of /list/ seems like a extremely poorly aimed
big hammer.
Can't we use wildcards to at least allow everything but the /since/
links? E.g. Disallow: /list/*/since/*. Is it because we're some less
common crawler doesn't implement wildcards at all?
Or slap rel=nofollow on links / add a meta tag preventing /since/ pages
from being indexed.
Yes, that'd not be perfect for the bigger lists, because there's no
"direct" way to get from the month's archive to all the month's emails
when paginated. But there's still the next/prev links. And it'd be much
better than what we have right now.
Greetings,
Andres Freund