Re: Request for feedback on hardware for a new database server

On Thu, Mar 17, 2011 at 7:51 PM, Oliver Charles
<postgresql-perf@ocharles.org.uk> wrote:
> Hello,
>
> At MusicBrainz we're looking to get a new database server, and are
> hoping to buy this in the next couple of days. I'm mostly a software
> guy, but I'm posting this on behalf of Rob, who's actually going to be
> buying the hardware. Here's a quote of what we're looking to get:
>
>    I'm working to spec out a bad-ass 1U database server with loads of
>    cores (12), RAM (24GB) and drives (4 SAS) in a hardware RAID-1,0
>    configuration:
>
>    1 * SuperMicro 2016R-URF, 1U, redundant power supply, 4 SATA/SAS
>    drive bays 2
>    2 * Intel Xeon X5650 Westmere 2.66GHz 12MB L3 Cache LGA 1366 95W
>    Six-Core Server Processor 2
>    2 * Crucial 24GB (3 x 4GB) DDR3 SDRAM ECC Registered DDR3 1333,
>    CT3KIT51272BV1339 1
>    1 * LSI MegaRAID SATA/SAS 9260-4i ($379) (linux support [1])
>    or
>    1 * HighPoint RocketRAID 4320 PCI-Express x8 ($429)
>    or
>    1 * Adaptec RAID 3405 controller ($354)
>    4 * Fujitsu MBA3147RC 147GB 15000 RPM
>
>    SuperMicro machines have treated us really well over time (better
>    than Dell or Sun boxes), so I am really happy to throw more money in
>    their direction.  Redundant power supplies seem like a good idea for
>    a database server.
>
>    For $400 more we can get hexa core processors as opposed to quad
>    core processors at 2.66Ghz. This seems like a really good deal --
>    any thoughts on this?
>
>    Crucial memory has also served us really well, so that is a
>    no-brainer.
>
>    The RAID controller cards are where I need to most feedback! Of the
>    LSI, Highpoint or Adaptec cards, which one is likely to have native
>    linux support that does not require custom drivers to be installed?
>    The LSI card has great specs at a great price point with Linux
>    support, but installing the custom driver sounds like a pain. Does
>    anyone have any experience with these cards?
>
>    We've opted to not go for SSD drives in the server just yet -- it
>    doesn't seem clear how well SSDs do in a driver environment.
>
>    That's it -- anyone have any feedback?
>
> Just a quick bit more information. Our database is certainly weighted
> towards being read heavy, rather than write heavy (with a read-only web
> service accounting for ~90% of our traffic). Our tables vary in size,
> with the upperbound being around 10mil rows.

It doesn't sound like SSD are a good fit for you -- you have small
enough data that you can easily buffer in RAM and not enough writing
to bottleneck you on the I/O side.  The #1 server building mistake is
focusing too much on cpu and not enough on i/o, but as noted by others
you should be ok with a decent raid controller with a bbu on it.  A
bbu will make a tremendous difference in server responsiveness to
sudden write bursts (like vacuum), which is particularly critical with
your whole setup being on a single physical volume.

Keeping your o/s and the db on the same LUN is a dangerous btw because
it can limit your ability to log in and deal with certain classes of
emergency situations.  It's possible to do a hybrid type setup where
you keep your o/s mounted on a CF or even a thumb drive(s) (most 1U
servers now have internal usb ports for exactly this purpose) but this
takes a certain bit of preparation and understanding what is sane to
do with flash..

My other concern with your setup is you might not have room for
expansion unless you have an unallocated pci-e slot in the back (some
1U have 1, some have 2).  With an extra slot, you can pop a sas hba in
the future attached to an enclosure if your storage requirements go up
significantly.

Option '2' is to go all out on the raid controller right now, so that
you have both internal and external sas ports, although these tend to
be much more expensive.  Option '3' is to just 2U now, leaving
yourself room for backplane expansion.

Putting it all together, I am not a fan of 1U database boxes unless
you are breaking the storage out -- there are ways you can get burned
so that you have to redo all your storage volumes (assuming you are
not using LVM, which I have very mixed feelings about) or even buy a
completely new server -- both scenarios can be expensive in terms of
downtime.

merlin