Обсуждение: Tyan Thunder MB for postgres server

Hi All,

I've read a fair bit in these lists about server configurations, and have
come to some conclusions as to the kind of system that I want to build.
However, I would like to hear from anyone with specific experience of this
config. before I go ahead and buy it:

The system is based on a Tyan Thunder K8S motherboard, dual opterons and 4GB
RAM, and the LSI dual channel raid option. OS will be RHEL AS 3 for AMD64

The Tyan/opteron options seems tobe quite popular as a postgres server, but
I'm not sure how many people are doing it all 64 bit, so my specific areas
of concern at the moment are:

1. 64bit linux driver support - how is it?

2. Is there a postgres 7.4.6 package available for this (ie AMD64) or will I
have to compile it myself?

Personally, I don't mind compiling it myself - have done it many times, but
since I have to write instructions for the system to be  completely re-built
in the case of a failure, I'd rather go with widely available packages as
much as possible.

I hope to hear from you,
regards
Iain

We use that exact configuration right now, except with an Adaptec card
and more RAM.

We used RHEL 3.0, then switched to Fedora core 2 64Bit as a test, since
this server was since placed into standby duties. Well, we needed to use
the server when the main server went into maintenance recently, and it
worked fantastically well with Fedora 64Bit.

You want to get the RPM source package for Postgresql and build the
RPM's instead of installing from source. This method builds the binary
RPM's for your platform which you can then install normally afterwards.

Warmest regards,
Ericson Smith
Tracking Specialist/DBA
+-----------------------+------------------------------+
| http://www.did-it.com | Need help tracking your paid |
| eric@did-it.com       | search campaigns?            |
| 516-255-0500          |        - Help is on the way! |
+-----------------------+------------------------------+



Iain wrote:

> Hi All,
>
> I've read a fair bit in these lists about server configurations, and
> have come to some conclusions as to the kind of system that I want to
> build. However, I would like to hear from anyone with specific
> experience of this config. before I go ahead and buy it:
>
> The system is based on a Tyan Thunder K8S motherboard, dual opterons
> and 4GB RAM, and the LSI dual channel raid option. OS will be RHEL AS
> 3 for AMD64
>
> The Tyan/opteron options seems tobe quite popular as a postgres
> server, but I'm not sure how many people are doing it all 64 bit, so
> my specific areas of concern at the moment are:
>
> 1. 64bit linux driver support - how is it?
>
> 2. Is there a postgres 7.4.6 package available for this (ie AMD64) or
> will I have to compile it myself?
>
> Personally, I don't mind compiling it myself - have done it many
> times, but since I have to write instructions for the system to be
> completely re-built in the case of a failure, I'd rather go with
> widely available packages as much as possible.
>
> I hope to hear from you,
> regards
> Iain
>
>
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if
> your
> joining column's datatypes do not match
>

Hi Ericson,

I'm planning on using the onboard LSI SCSI controller, and have read alot
about poor IO performance on Dells using LSI. Then  again, most of the talk
about Adaptec hasn't been all that complimentary either as I saw it ;-) It
seems that there is more to it than the name of the chip vendor. I havn't
heard anything bad about the Tyan opteron based boards in any configuration
yet though.

Thanks for your feedback,
regards
Iain


----- Original Message -----
From: "Ericson Smith" <eric@did-it.com>
To: <pgsql-admin@postgresql.org>
Sent: Tuesday, December 14, 2004 12:11 AM
Subject: Re: [ADMIN] Tyan Thunder MB for postgres server


> We use that exact configuration right now, except with an Adaptec card
> and more RAM.
>
> We used RHEL 3.0, then switched to Fedora core 2 64Bit as a test, since
> this server was since placed into standby duties. Well, we needed to use
> the server when the main server went into maintenance recently, and it
> worked fantastically well with Fedora 64Bit.
>
> You want to get the RPM source package for Postgresql and build the
> RPM's instead of installing from source. This method builds the binary
> RPM's for your platform which you can then install normally afterwards.
>
> Warmest regards,
> Ericson Smith
> Tracking Specialist/DBA
> +-----------------------+------------------------------+
> | http://www.did-it.com | Need help tracking your paid |
> | eric@did-it.com       | search campaigns?            |
> | 516-255-0500          |        - Help is on the way! |
> +-----------------------+------------------------------+
>
>
>
> Iain wrote:
>
>> Hi All,
>>
>> I've read a fair bit in these lists about server configurations, and
>> have come to some conclusions as to the kind of system that I want to
>> build. However, I would like to hear from anyone with specific
>> experience of this config. before I go ahead and buy it:
>>
>> The system is based on a Tyan Thunder K8S motherboard, dual opterons
>> and 4GB RAM, and the LSI dual channel raid option. OS will be RHEL AS
>> 3 for AMD64
>>
>> The Tyan/opteron options seems tobe quite popular as a postgres
>> server, but I'm not sure how many people are doing it all 64 bit, so
>> my specific areas of concern at the moment are:
>>
>> 1. 64bit linux driver support - how is it?
>>
>> 2. Is there a postgres 7.4.6 package available for this (ie AMD64) or
>> will I have to compile it myself?
>>
>> Personally, I don't mind compiling it myself - have done it many
>> times, but since I have to write instructions for the system to be
>> completely re-built in the case of a failure, I'd rather go with
>> widely available packages as much as possible.
>>
>> I hope to hear from you,
>> regards
>> Iain
>>
>>
>>
>>
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 9: the planner will ignore your desire to choose an index scan if
>> your
>> joining column's datatypes do not match
>>
>


--------------------------------------------------------------------------------


>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
>      subscribe-nomail command to majordomo@postgresql.org so that your
>      message can get through to the mailing list cleanly
>

I've used a slew of LSI controllers (22915A, integrated 53C1010,
22320-R, MegaRAID 320-1) and they've all performed w/o issues. Now I
have had some hardware die though. One was probably our fault -- during
an attempted upgrade, we probably weren't careful enough with the
22320-R (cramped #@#@$! rackmounts) and after we reinstalled it, it just
never worked right again.

Otherwise, if you leave it alone, they work pretty good. I've got a 4
15K disks (software RAID10) attached to a 22915A in a 2x Opteron 244
server running FC2 64-bit.  This machine is deadly fast; every so often,
iostat will show peak transfer rates of 160MB/s. (RAID
cage+rackmount problems stopped us from upgrading to hardware
raid+battery -- we'll tackle this again when we get a 4U case.)


Iain wrote:

> Hi Ericson,
>
> I'm planning on using the onboard LSI SCSI controller, and have read
> alot about poor IO performance on Dells using LSI. Then  again, most of
> the talk about Adaptec hasn't been all that complimentary either as I
> saw it ;-) It seems that there is more to it than the name of the chip
> vendor. I havn't heard anything bad about the Tyan opteron based boards
> in any configuration yet though.

I've used LSI/MegaRAID controllers with the 2600 series servers with no
problems whatsoever in terms of performance.  With battery backed cache
the machine never went above 0.05 load factor in production, and could
handle pgbench with 300 to 600 tps under various settings.  Loading a 10
gig data set with fks and indexes took about 5 or 6 minutes.

On Mon, 2004-12-13 at 19:32, Iain wrote:
> Hi Ericson,
>
> I'm planning on using the onboard LSI SCSI controller, and have read alot
> about poor IO performance on Dells using LSI. Then  again, most of the talk
> about Adaptec hasn't been all that complimentary either as I saw it ;-) It
> seems that there is more to it than the name of the chip vendor. I havn't
> heard anything bad about the Tyan opteron based boards in any configuration
> yet though.
>
> Thanks for your feedback,
> regards
> Iain
>
>
> ----- Original Message -----
> From: "Ericson Smith" <eric@did-it.com>
> To: <pgsql-admin@postgresql.org>
> Sent: Tuesday, December 14, 2004 12:11 AM
> Subject: Re: [ADMIN] Tyan Thunder MB for postgres server
>
>
> > We use that exact configuration right now, except with an Adaptec card
> > and more RAM.
> >
> > We used RHEL 3.0, then switched to Fedora core 2 64Bit as a test, since
> > this server was since placed into standby duties. Well, we needed to use
> > the server when the main server went into maintenance recently, and it
> > worked fantastically well with Fedora 64Bit.
> >
> > You want to get the RPM source package for Postgresql and build the
> > RPM's instead of installing from source. This method builds the binary
> > RPM's for your platform which you can then install normally afterwards.
> >
> > Warmest regards,
> > Ericson Smith
> > Tracking Specialist/DBA
> > +-----------------------+------------------------------+
> > | http://www.did-it.com | Need help tracking your paid |
> > | eric@did-it.com       | search campaigns?            |
> > | 516-255-0500          |        - Help is on the way! |
> > +-----------------------+------------------------------+
> >
> >
> >
> > Iain wrote:
> >
> >> Hi All,
> >>
> >> I've read a fair bit in these lists about server configurations, and
> >> have come to some conclusions as to the kind of system that I want to
> >> build. However, I would like to hear from anyone with specific
> >> experience of this config. before I go ahead and buy it:
> >>
> >> The system is based on a Tyan Thunder K8S motherboard, dual opterons
> >> and 4GB RAM, and the LSI dual channel raid option. OS will be RHEL AS
> >> 3 for AMD64
> >>
> >> The Tyan/opteron options seems tobe quite popular as a postgres
> >> server, but I'm not sure how many people are doing it all 64 bit, so
> >> my specific areas of concern at the moment are:
> >>
> >> 1. 64bit linux driver support - how is it?
> >>
> >> 2. Is there a postgres 7.4.6 package available for this (ie AMD64) or
> >> will I have to compile it myself?
> >>
> >> Personally, I don't mind compiling it myself - have done it many
> >> times, but since I have to write instructions for the system to be
> >> completely re-built in the case of a failure, I'd rather go with
> >> widely available packages as much as possible.
> >>
> >> I hope to hear from you,
> >> regards
> >> Iain
> >>
> >>
> >>
> >>
> >>
> >>
> >> ---------------------------(end of broadcast)---------------------------
> >> TIP 9: the planner will ignore your desire to choose an index scan if
> >> your
> >> joining column's datatypes do not match
> >>
> >
>
>
> --------------------------------------------------------------------------------
>
>
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 3: if posting/reading through Usenet, please send an appropriate
> >      subscribe-nomail command to majordomo@postgresql.org so that your
> >      message can get through to the mailing list cleanly
> >
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

Thanks to all for your feedback on this.

I'm looking forward to getting my hands on the system.

It seems that the battery backed cache is an important factor, though I
havn't found any information specifically concerning this on the tyan site.
I can't tell if it's an option or standard with the integrated SCSI
controller.

I don't have much experience with building this kind of server (i'm more on
the development side) but I know basically what I want and that is RAID 0+1
and battery backed cache. The target OS is RHEL ES 3 (AMD64 platform) and I
guess that it doesn't matter if the RAID is hardware or software (or a
combination of both?) but if you think I've missed anything, please let me
know.

Regards
Iain

Iain wrote:

> Thanks to all for your feedback on this.
>
> I'm looking forward to getting my hands on the system.
>
> It seems that the battery backed cache is an important factor, though I
> havn't found any information specifically concerning this on the tyan
> site. I can't tell if it's an option or standard with the integrated
> SCSI controller.

RAID may be an option with your motherboard. LSI Logic has a MegaRAID
320-0 which interfaces with integrated MB SCSI.

http://www.lsilogic.com/products/megaraid/scsi_320_0.html

They don't list any Tyan's as "certified" so I dunno. Perhaps you should
look into a Tyan w/o integrated SCSI and just get the MegaRAID 320-1 or
320-2 to avoid any possible issues.

> I don't have much experience with building this kind of server (i'm more
> on the development side) but I know basically what I want and that is
> RAID 0+1 and battery backed cache. The target OS is RHEL ES 3 (AMD64
> platform) and I guess that it doesn't matter if the RAID is hardware or
> software (or a combination of both?) but if you think I've missed
> anything, please let me know.

Hardware RAID + battery lets you turn write caching on. For a DB with
lots of updates, it can make a big difference.

Hi William,

> They don't list any Tyan's as "certified" so I dunno. Perhaps you should
> look into a Tyan w/o integrated SCSI and just get the MegaRAID 320-1 or
> 320-2 to avoid any possible issues.

Upon further investigation, I think this the way to go, specificaly I'llbe
recommending the 320-2 as I plan to put the WAL and DB on seperate channels
and partitions (as opposed to to putting them on the same logical partition
split over the two channels).

> Hardware RAID + battery lets you turn write caching on. For a DB with
> lots of updates, it can make a big difference.

And we need all the help we can get, right? ;-)

Thanks for pointing me in the right direction there.

regards
Iain

Iain wrote:
> Upon further investigation, I think this the way to go, specificaly
> I'llbe recommending the 320-2 as I plan to put the WAL and DB on
> seperate channels and partitions (as opposed to to putting them on the
> same logical partition split over the two channels).

SOmething to think about. Let's suppose a channel/cable completely dies.
How would you protect against it? Split a logical mirror device over 2
channels.

Another trick I've started doing with my MegaRAID setups is mirroring in
hardware but striping in software. To do 100% hardware RAID-10 MegaRAID
cards, all drives must have the same geometry. Not a big deal when
you're first configuring a server but a few years down the line when a
drive dies and you need a replacement, it can drive you up the wall. I
had a set of IBM 36GBs that needed a replacement -- couldn't find any
old stock so I decided to get the newest IBM (Hitachi) 36GBs -- NOT THE
SAME GEOMETRY!@!@!#!$##

After learning that lesson, I did some testing with mirroring. LSI is
much less stringent for mirroring. I was able to pair a Seagate + IBM
and still get a valid mirror drive. I then use Linux software RAID to
stripe 2 mirror volumes together.

Hi William,

> SOmething to think about. Let's suppose a channel/cable completely dies.
> How would you protect against it? Split a logical mirror device over 2
> channels.

This effectively implements RAID 0+1, right? RAID 1 (mirroring) over RAID 0
striped volumes. I can certainly see your point regarding the redundancy of
the controller channels, but my understanding is that (apart from that) RAID
0+1 is less robust that RAID 10 regarding disk failures. Presuming that the
system will continue to operate even in the event of 1 channel failure, it's
still not a clear choice. Does that seem like a reasonable assessment?

> Another trick I've started doing with my MegaRAID setups is mirroring in
> hardware but striping in software.

Yeah, that is a good point. I havn't decided either way but I consider that
a viable option.

If you were building this system now, and want the option of buying the same
disks in 3 years time, do you think it would be a bad idea to go for the
~40GB size? Maybe the next size up would be better, though we don't actually
need the extra space.

If you have any specific recommendations (for or against) specific
drives/manufacturers, please let me know.

Also, someone asked me what happens if one of the CPUs fails on this system,
will the system continue to operate on 1 CPU. I havn't really considered
this, and have never read anything either way, so my assumption is "no, it
won't". Any comment?

Thanks again
Iain

Iain wrote:
> Hi William,
>
>> SOmething to think about. Let's suppose a channel/cable completely
>> dies. How would you protect against it? Split a logical mirror device
>> over 2 channels.
>
>
> This effectively implements RAID 0+1, right? RAID 1 (mirroring) over
> RAID 0 striped volumes. I can certainly see your point regarding the
> redundancy of the controller channels, but my understanding is that
> (apart from that) RAID 0+1 is less robust that RAID 10 regarding disk
> failures. Presuming that the system will continue to operate even in the
> event of 1 channel failure, it's still not a clear choice. Does that
> seem like a reasonable assessment?

I'm not sure why people say one is better than the other. Both will
survive the loss of 2 drives -- they're just different drives.

RAID 0+1: A(1m1) s B(1m1) <-- any drive on A and any drive on B
RAID 10:  A(1s1) m B(1s1) <-- both drives on A or both drives on B

Either way, I think you can do both across 2 channels to be redundant
for a channel/cable failing.

RAID 0+1: C1-m-C2 s C1-m-C2
RAID 10:  C1-s-C1 m C2-s-C2


>> Another trick I've started doing with my MegaRAID setups is mirroring
>> in hardware but striping in software.
>
>
> Yeah, that is a good point. I havn't decided either way but I consider
> that a viable option.

You can probably do the same hardware + software trick for RAID10.
Except after OS crashes, Linux+Solaris spends quite a bit of time
resyncing mirrors and it looks like the MegaRAID controller is a bit
smarter in knowing when to resync.


> If you were building this system now, and want the option of buying the
> same disks in 3 years time, do you think it would be a bad idea to go
> for the ~40GB size? Maybe the next size up would be better, though we
> don't actually need the extra space.

Always get more space than you need if you can afford it because it's a
pain in the ass adding more disk space. Especially on the boot drives.


> Also, someone asked me what happens if one of the CPUs fails on this
> system, will the system continue to operate on 1 CPU. I havn't really
> considered this, and have never read anything either way, so my
> assumption is "no, it won't". Any comment?

Doing a google search for "hot swap cpu", I see they've added something
into Linux but you'd still need hardware support.

On Thu, 16 Dec 2004 11:18:37 +0900, Iain <iain@mst.co.jp> wrote:
> Also, someone asked me what happens if one of the CPUs fails on this system,
> will the system continue to operate on 1 CPU. I havn't really considered
> this, and have never read anything either way, so my assumption is "no, it
> won't". Any comment?

You are correct. That kind of functionality can't be found on standard
x86 gear (sans the $150K NEC redundant behemoth).

Regards,
aaron.glenn

Thanks again for your valuable input William.

> I'm not sure why people say one is better than the other. Both will
> survive the loss of 2 drives -- they're just different drives.

> RAID 0+1: A(1m1) s B(1m1) <-- any drive on A and any drive on B
> RAID 10:  A(1s1) m B(1s1) <-- both drives on A or both drives on B

I'm wondering if we have a terminology problem here. As I understand it the
configs are the reverse of what you said. I only checked  www.bytepile.com:
so I hope they're not wrong! I quote:

"RAID 0+1 is implemented as a mirrored array whose segments are RAID 0"

"RAID 10 is implemented as a striped array whose segments are RAID 1"

This is the quote that interested me "RAID 0+1 is NOT to be confused with
RAID 10. A single drive failure will cause the whole array to become, in
essence, a RAID Level 0 array" and "Under certain circumstances, RAID 10
array can sustain multiple simultaneous drive failures".

As bytepile has it, failure of 1 disk in 0+1 leaves you with just RAID 0 so
one more failure on the other pair and your data is gone. On the other hand,
failure of 1 disk in raid 10 leaves you with a working raid 1 that can
sustain a second failure.

> Either way, I think you can do both across 2 channels to be redundant for
> a channel/cable failing.
>
> RAID 0+1: C1-m-C2 s C1-m-C2
> RAID 10:  C1-s-C1 m C2-s-C2

Gaaa! I'll have to ponder that one. C1-m-C2 s C1-m-C2 sounds good in theory.
Presumably to be practical it would require hardware and maybe software
support, so the machine continues to operate despite the failure (and
hopefully tells you about it). It would interesting to test it anyway.

> Always get more space than you need if you can afford it because it's a
> pain in the ass adding more disk space. Especially on the boot drives.

Very true.

> Doing a google search for "hot swap cpu", I see they've added something
> into Linux but you'd still need hardware support.

I'm sure that if the Tyan board had that support they'd be making a fuss of
it :-) I'll have a look for "hot swap cpu" too though, just for interests
sake.

High redundancy is becoming more accessible I think, but it still takes a
lot more money than your average PC server motherboard costs to get it ;-)
something to look forward to anyway.

Cheers,
Iain

Hi Aaron,

> You are correct. That kind of functionality can't be found on standard
> x86 gear (sans the $150K NEC redundant behemoth).

I reckon that the day is coming though, and I expect Linux will ready when
the hardware arrives :-)

cheers
Iain

Iain wrote:

> As bytepile has it, failure of 1 disk in 0+1 leaves you with just RAID 0
> so one more failure on the other pair and your data is gone. On the
> other hand, failure of 1 disk in raid 10 leaves you with a working raid
> 1 that can sustain a second failure.

What they're saying is in the case of (AsB) m (CsD) -- if A fails, they
no longer count B as part of the array and no longer part of the
possible drives that can fail. Sorta like the "no one hears a tree fall,
did it fall" scenario.

I personally disagree with that theory. B is still part of the array.
Pop in a new drive and the array is ready to start resync (CsD) -->
(AsB). You still have a 1/3 chance in surviving another drive failure as
long as B is the one that dies.

Although now that I think about it, RAID10 is more resillient because
the odds are survival after 1 failure is 2/3. In the case of (AmB) s
(CmD), if A fails, you can survive C failing or D failing.

On Wed, Dec 15, 2004 at 11:14:03PM -0800, William Yu wrote:
>
> I'm not sure why people say one is better than the other. Both will
> survive the loss of 2 drives -- they're just different drives.

Partly, I think, people who've used both hate 0+1 because of the
recovery cost.  In most 0+1 arrangements (I'm aware of none which
don't do this), you have to re-sync the entire thing in case you lose
even one drive.  Performance really suffers at that point.

A

--
Andrew Sullivan  | ajs@crankycanuck.ca
I remember when computers were frustrating because they *did* exactly what
you told them to.  That actually seems sort of quaint now.
        --J.D. Baldwin