On 24 dec 2008, at 10.24, Stefan Kaltenbrunner
<stefan@kaltenbrunner.cc> wrote:
> Magnus Hagander wrote:
>> FYI - I've committed a new version of the URL redirector for
>> downloads.
>> The old version was being used for linkfilter-breakthrough to
>> distribute
>> viruses :-(
>> Since I was hacking around that code anyway, I didn't just add a
>> filter
>> to it, but changed around how it works a bit. Apart from it no longer
>> being possible to use it to break through stupid linkblockers, it has
>> also made the URLs easier to read and copy/paste, and we're also
>> storing
>> the logging information in a way that's much easier to analyze than
>> before.
>> Do keep your eyes open for bugs, of course :-)
>
> this change broke most of the website replication code and is close
> to running out some of the website mirrors out of diskspace. It
> seems that the mirror script is now copying tons of /redir/
> <mirrorid> directories to the slaves and some of them contain
> indvidual copies of the full source tarball for all active releases.
> This causes both disk-usage related issues as well as very long sync-
> times between wwwmaster and the slaves...
> I don't have time to look into that more closely now so it would ge
> good if somebody else could
Oh shit...
It shouldn't crawl explicit links to wwwmaster, I thought :( perhaps
some place is forgetting to make it explicit?
If not, then just making it exclude everything under /redir/ wyen
mirroring should do the trick.
Unfortunately it'll be a while before I can look at it, so I'd
appreciate if yet someone else could!
/Magnus