Обсуждение: SAN/NAS options

От:
Charles Sprickman
Дата:

Hello all,

It seems that I'm starting to outgrow our current Postgres setup.  We've
been running a handful of machines as standalone db servers.  This is all
in a colocation environment, so everything is stuffed into 1U Supermicro
boxes.  Our standard build looks like this:

Supermicro 1U w/SCA backplane and 4 bays
2x2.8 GHz Xeons
Adaptec 2015S "zero channel" RAID card
2 or 4 x 73GB Seagate 10K Ultra 320 drives (mirrored+striped)
2GB RAM
FreeBSD 4.11
PGSQL data from 5-10GB per box

Recently I started studying what we were running up against in our nightly
runs that do a ton of updates/inserts to prep things for the tasks the db
does during the business day (light mix of selects/inserts/updates).
While we have plenty of disk bandwidth (according to bonnie), we are
really dying on IOPS.  I'm guessing this is a mix of a rather anemic RAID
controller (ever notice how adaptec doesn't publish any real
performance specs on raid cards?) and having only two or four spindles
(effectively 1 or 2 on writes).

So that's where we are...

I'm new to the whole SAN thing, but did recently pick up a few used NetApp
shelves and a Fibre Channel RAID HBA (Mylex ExtremeRAID 3000, also used)
to toy with.  I started wondering if I could put something together to
both get our storage on one set of boxes and allow me to get data striped
across more drives.  Our budget is not huge and we are not adverse to
getting used gear where appropriate.

What do you folks recommend?  I'm just starting to look at what's out
there for SANs and NAS, and from what I've seen, our options are:

NetApp Filers - the pluses with these are that if we use NFS, we don't
have to worry about either large filesystem support in FreeBSD (2TB
practical limit), or limitation on "growing" partitions as the NetApp just
deals with that.  I also understand these make backups a bit simpler.  I
have a great, trusted, spare-stocking source for these.

Apple X-Serve RAID - well, it's pretty cheap.  Honestly, that's all I know
about it - they don't talk about IOPS numbers, and I have no idea what
lurks in that box as a RAID controller.

SAN box w/integrated RAID - it seems like this might not be a good choice
since the RAID hardware in the box may be where I hit any limits.  I also
imagine I'm probably overpaying for some OEM RAID controller integrated
into the box.  No idea where to look for used gear.

SAN box, JBOD - this seems like it might be affordable as well.  A few big
shelves full of drives a SAN "switch" to plug all the shelves and hosts
into and a FC RAID card in each host.  No idea where to look for used gear
here either.

You'll note that I'm being somewhat driven by my OS of choice, FreeBSD.
Unlike Solaris or other commercial offerings, there is no nice volume
management available.  While I'd love to keep managing a dozen or so
FreeBSD boxes, I could be persuaded to go to Solaris x86 if the volume
management really shines and Postgres performs well on it.

Lastly, one thing that I'm not yet finding in trying to educate myself on
SANs is a good overview of what's come out in the past few years that's
more affordable than the old big-iron stuff.  For example I saw some brief
info on this list's archives about the Dell/EMC offerings.  Anything else
in that vein to look at?

I hope this isn't too far off topic for this list.  Postgres is the
main application that I'm looking to accomodate.  Anything else I can do
with whatever solution we find is just gravy...

Thanks!

Charles


От:
Charles Sprickman
Дата:

On Wed, 14 Dec 2005, Charles Sprickman wrote:

[big snip]

The list server seems to be regurgitating old stuff, and in doing so it
reminded me to thank everyone for their input.  I was kind of waiting to
see if anyone who was very pro-NAS/SAN was going to pipe up, but it looks
like most people are content with per-host storage.

You've given me a lot to go on...  Now I'm going to have to do some
research as to real-world RAID controller performance.  It's vexing (to
say the least) that most vendors don't supply any raw throughput or TPS
stats on this stuff...

Anyhow, thanks again.  You'll probably see me back here in the coming
months as I try to shake some mysql info out of my brain as our pgsql DBA
gets me up to speed on pgsql and what specifically he's doing to stress
things.

Charles

> I hope this isn't too far off topic for this list.  Postgres is the main
> application that I'm looking to accomodate.  Anything else I can do with
> whatever solution we find is just gravy...
>
> Thanks!
>
> Charles
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>

От:
Alex Stapleton
Дата:

>
>> I hope this isn't too far off topic for this list.  Postgres is
>> the main application that I'm looking to accomodate.  Anything
>> else I can do with whatever solution we find is just gravy...
> You've given me a lot to go on...  Now I'm going to have to do some
> research as to real-world RAID controller performance.  It's vexing
> (to say the least) that most vendors don't supply any raw
> throughput or TPS stats on this stuff...

One word of advice. Stay away from Dell kit. The PERC 4 controllers
they use don't implement RAID 10 properly. It's RAID 1 + JBOD array.
It also has generally dismal IOPS performance too. You might get away
with running software RAID, either in conjunction with, or entirely
avoiding the card.

От:
"Luke Lonergan"
Дата:

Charles,

On 12/20/05 9:58 PM, "Charles Sprickman" <> wrote:

> You've given me a lot to go on...  Now I'm going to have to do some
> research as to real-world RAID controller performance.  It's vexing (to
> say the least) that most vendors don't supply any raw throughput or TPS
> stats on this stuff...

Take a look at this:
  http://www.wlug.org.nz/HarddiskBenchmarks

> Anyhow, thanks again.  You'll probably see me back here in the coming
> months as I try to shake some mysql info out of my brain as our pgsql DBA
> gets me up to speed on pgsql and what specifically he's doing to stress
> things.

Cool!

BTW - based on the above benchmark page, I just immediately ordered 2 x of
the Areca 1220 SATA controllers (
http://www.areca.com.tw/products/html/pcie-sata.htm) so that we can compare
them to the 3Ware 9550SX that we've been using.  The 3Ware controllers have
been super fast on sequential access, but I'm concerned about their random
IOPs.  The Areca's aren't as popular, and there's consequently less volume
of them, but people who use them rave about them.

- Luke



От:
Charles Sprickman
Дата:

On Sat, 14 Jan 2006, Luke Lonergan wrote:

> Charles,
>
> On 1/14/06 6:37 PM, "Charles Sprickman" <> wrote:
>
>> I'm vaguely considering pairing these two devices:
>>
>> http://www.areca.us/products/html/products.htm
>>
>> That's an Areca 16 channel SATA II (I haven't even read up on what's new
>> in SATA II) RAID controller with an optional U320 SCSI daughter card to
>> connect to the host(s).
>
> I'm confused - SATA with a SCSI daughter card? Where does the SCSI go?

Bad ASCII diagram follows (D=disk, C=controller H=host):

    SATA   ____
D -------|    |  SCSI   ________
D -------| C  |--------|    H   |
D -------|    |        |________|
D -------|____|

(etc. up
to 16
drives)

The drives and the controller go in the Chenbro case.  U320 SCSI from the
RAID controller in the Chenbro case to the 1U server.

C

От:
"Luke Lonergan"
Дата:

Charles,

On 1/14/06 7:23 PM, "Charles Sprickman" <> wrote:

> The drives and the controller go in the Chenbro case.  U320 SCSI from the
> RAID controller in the Chenbro case to the 1U server.

Thanks for the explanation - I didn't click on your Areca link until now,
thinking it was a generic link to their products page.

Looks great - I think this might do better than the SATA -> FC products
because of the use of faster processors, but I'd keep my expectations low
until we see some performance data on it.

We've had some very poor experiences with Fibre Channel attach SATA disk
controllers.  A large vendor of same ultimately concluded that they will no
longer recommend them for database use because of the terrible performance
of their unit.  We ended up with a 110MB/s bottleneck on the controller when
using 200MB/s FC connections.

With the dual U320 attach and 16 drives, you should be able to saturate the
SCSI busses at about 600MB/s.  It would be great if you could post your I/O
results here!

- Luke