Re: Yet another infrastructure problem

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160


Stefan wrote:

>> 2) Battling over resources and causing one jail to affect another

> that one has happened - but only one or two times over the last few
> years so I'm not convinced it is a real issue rather than an isolated
> incident.

I think this happends more than you realize. Isn't the jabber service
still causing problems now? Wasn't the wiki recently affected by something
else? Who knows how often it happens to a lesser extent? It's only the
extreme cases that cause notices to be sent to this list.

>> 3) Hardware problems that affect more than one jail

> the very same would happen if we used some sort of full virtualization
> technology so I'm not sure I see the point. Or are you actively
> proposing we should request and run 40+ physical servers in the future ?
> I don't think that would be sensible in any way (both from a resource
> wasting pov and the administrative overhead - and we don't have that
> many boxes either).

No, not 40+, but having the small handful of important services distributed
on separate boxes/data centers would be a good idea. Specifically, the
archives, search, website, wiki, cvs, and mailing lists should ideally all be
on different servers, to minimize the impact on the project as a whole
when something goes down.

>> * One way around problems like this is to mirror the services.
>> That may involve load balancing, DNS tricks, database replication,
>> and other assorted goodies. It may be difficult, but it's something
>> I'd like to at least start us talking about.

> the low hanging fruit in that regard has already been taken (have you
> seen the static part of website being down in the last few years?) -

No, I'm completely happy with the static part of the website.

> most of the other services are much much harder to operate in a
> loadbalanced (or master-master) setup or doing it seems simply overkill.
> Furthermore I don't think that just making services more complex (as in
> redundant) will necessarily result in better availability. Howver I
> aknowledge that we can improve in some areas (like wiki authentication).

Er...how do you figure redundant services do not necessarily result in better
availability? That's kind of its point - and we certainly don't have anywhere
near 100% uptime for practically any part of our infrastructure. I do recognize
there is a complexity tradefoff to be made, so perhaps only some (or none)
of the services need that tradeoff to be made. However, I consider it a valid
point to be raised. This goes a little to disaster recovery as well, so perhaps
some of the services (e.g. cvs) are already mirrored in some fashion, and all
we need to do is to tweak some things?

>> * As much as I love the concept of BSD (and I might even be running it
>> at home if it didn't always coredump while installing on my laptop), we
>> should realize that the there are many people in our community who are
>> really, really good with Linux. Many of the people on the PG lists do
>> Linuxy support as their dayjob. I'm not saying we should dump BSD, but
>> I'm dismayed to see the resistance given to adding non-BSD boxes to our
>> mix.

> Not against that idea in general (and we already have a fair share of
> linux boxes too) how would linux solve any of the issues you mentioned ?
> All of the linux distributions had their fair share of "breaking stuff
> with security/point updates/upgrades" and if hardware breaks it doesn't
> matter if we run BSD, Linux or Windows.

It wouldn't solve any of the above issues, which is why it was the last
bullet point. As Robert points out, we could just switch to Sun's Solaris,
then we wouldn't have any problems. Look how well MySQL is going under their
watch! :)

Joshua Drake wrote:

> PostgreSQL.Org uses a FreeBSD architecture. To my knowledge there are
> only two exceptions to this, one of which will go away by the end of the
> month. Don't ask for linux -- you aren't going to get it.

> We use jails. Deal with it.

We are dealing with it, that's one of the big problems.

> I use to buy into the argument of ... if we had Linux more people would
> be willing to help. That argument is crap. People will help if they want
> to help. They will learn what they need to help. Those that say, "if you
> were running linux I would help" have a good heart but aren't people
> that are really going to help in the long run anyway.

That counter-argument is crap as well. People will 'learn what they need to
help'? This is a volunteer project, so the more barriers we put in front of
people, the less that will get done. While homogeneity of servers can be a
good thing from a sysadmin perspective, expanding the pool of potential help
can be as well.

> So can we just put on the Wiki that this is the way it is? That way the
> next time it comes up, we just point.

Next time can we send a message to -announce and -general letting people
know the website, cvs, wiki, and pgadmin are going to be down? I think that
was one of the most annoying aspects of this whole incident.


- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200810292113
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAkkJDBEACgkQvJuQZxSWSsgMFQCgt51u2F4c/7TrSaVAO79Y293+
HbEAn1dM6owdqWZK0Ey06BzX9u56e6U8
=J3av
-----END PGP SIGNATURE-----