Re: Yet another infrastructure problem

Greg Sabino Mullane wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: RIPEMD160
> 
> 
>>> People have been complaining on IRC that nothing can be
>>> downloaded from our site, as the mirror-picking script throws
>>> an internal error.
>>> When are we going to fix our infrastructure properly?
> 
>> As Stefan has already posted on this very list, he is performing
>> maintenance on that machine in order to move it to new hardware.
> 
> I understand that, but I think this project is big enough, and
> important enough, and has enough smart people involved in it,
> that things like this should just not happen. Some thoughts, in
> order of descending importance to the matter at hand:
> 
> * Why do we have so many eggs in one basket? I know that "jails"
> allows us to have many subdomains/services on one physical box,
> but we've seen three problems with the concept lately:
> 
> 1) Global software updates that breaks things in all jails

we need to do software upgrades once in a while because OSes reach their 
EOL date (and therefor loose security support). Softwareupdates tend to 
break stuff and OSes are more complex than a single application so we 
have to expect some issues.
Security/Feature upgrades of userspace apps are obviously only affecting 
a single jail.

> 2) Battling over resources and causing one jail to affect another

that one has happened - but only one or two times over the last few 
years so I'm not convinced it is a real issue rather than an isolated 
incident.

> 3) Hardware problems that affect more than one jail

the very same would happen if we used some sort of full virtualization 
technology so I'm not sure I see the point. Or are you actively 
proposing we should request and run 40+ physical servers in the future ?
I don't think that would be sensible in any way (both from a resource 
wasting pov and the administrative overhead - and we don't have that 
many boxes either).

> 
> * One way around problems like this is to mirror the services.
> That may involve load balancing, DNS tricks, database replication,
> and other assorted goodies. It may be difficult, but it's something
> I'd like to at least start us talking about.

the low hanging fruit in that regard has already been taken (have you 
seen the static part of website being down in the last few years?) - 
most of the other services are much much harder to operate in a 
loadbalanced (or master-master) setup or doing it seems simply overkill.
Furthermore I don't think that just making services more complex (as in 
redundant) will necessarily result in better availability. Howver I 
aknowledge that we can improve in some areas (like wiki authentication).

> 
> * As much as I love the concept of BSD (and I might even be running it
> at home if it didn't always coredump while installing on my laptop), we
> should realize that the there are many people in our community who are
> really, really good with Linux. Many of the people on the PG lists do
> Linuxy support as their dayjob. I'm not saying we should dump BSD, but
> I'm dismayed to see the resistance given to adding non-BSD boxes to our
> mix.

Not against that idea in general (and we already have a fair share of 
linux boxes too) how would linux solve any of the issues you mentioned ? 
All of the linux distributions had their fair share of "breaking stuff 
with security/point updates/upgrades" and if hardware breaks it doesn't 
matter if we run BSD, Linux or Windows.



Stefan