Обсуждение: raid10 write performance

От:
Grzegorz Jaśkiewicz
Дата:

Hi folks,

is there a general problem with raid10 performance postgresql on it?
We see very low performance on writes (2-3x slower than on less
performant servers). I wonder if it is solely problem of raid10
configuration, or if it is postgresql's thing.

Would moving WAL dir to separate disk help potentially ?

We're running centos 5.4, and server config is:

x3550 M2, xeon 4c e5530 2.4ghz , 6GB of ram
disks: ibm 300gb 2.5 SAS

raid: serveRAID M5014 SAS/SATA controller

strip size is the default 128k


thanks.

--
GJ

От:
Justin Graf
Дата:

On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote:
> Hi folks,
>
> is there a general problem with raid10 performance postgresql on it?
> We see very low performance on writes (2-3x slower than on less
> performant servers). I wonder if it is solely problem of raid10
> configuration, or if it is postgresql's thing.
>

RAID 10 is the commonly suggested layout for DB's as its performance to
redundancy is good.
The question that begs to be ask is what is the IO layout on the other
servers your comparing against.


> Would moving WAL dir to separate disk help potentially ?
>

Yes it can have a big impact.
http://www.westnet.com/~gsmith/content/postgresql/TuningPGWAL.htm
http://wiki.postgresql.org/wiki/Performance_Analysis_Tools

http://wiki.postgresql.org/wiki/Performance_Optimization


> We're running centos 5.4, and server config is:
>
> x3550 M2, xeon 4c e5530 2.4ghz , 6GB of ram
> disks: ibm 300gb 2.5 SAS
>
> raid: serveRAID M5014 SAS/SATA controller
>
> strip size is the default 128k
>
>
> thanks.
>
>



All legitimate Magwerks Corporation quotations are sent in a .PDF file attachment with a unique ID number generated by
ourproprietary quotation system. Quotations received via any other form of communication will not be honored. 

CONFIDENTIALITY NOTICE: This e-mail, including attachments, may contain legally privileged, confidential or other
informationproprietary to Magwerks Corporation and is intended solely for the use of the individual to whom it
addresses.If the reader of this e-mail is not the intended recipient or authorized agent, the reader is hereby notified
thatany unauthorized viewing, dissemination, distribution or copying of this e-mail is strictly prohibited. If you have
receivedthis e-mail in error, please notify the sender by replying to this message and destroy all occurrences of this
e-mailimmediately. 
Thank you.

От:
Greg Smith
Дата:

Grzegorz Jaśkiewicz wrote:
> raid: serveRAID M5014 SAS/SATA controller
>

Do the "performant servers" have a different RAID card?  This one has
terrible performance, and could alone be the source of your issue.  The
ServeRAID cards are slow in general, and certainly slow running RAID10.

--
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
   www.2ndQuadrant.us


От:
Karl Denninger
Дата:

Justin Graf wrote:
On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote: 
Would moving WAL dir to separate disk help potentially ?     
Yes it can have a big impact.
WAL on a separate spindle will make a HUGE difference in performance.  TPS rates frequently double OR BETTER with WAL on a dedicated spindle.

Strongly recommended.

Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL is on a different physical disk.  Snapshotting the data disk where WAL is on a separate spindle and backing it up **WILL NOT WORK** and **WILL** result in an non-restoreable backup.

The manual discusses this but it's easy to miss.... don't or you'll get a NASTY surprise if something goes wrong.....

-- Karl
От:
Dave Crooke
Дата:

Of course, no backup strategy is complete without testing a full restore onto bare hardware :-)

On Tue, Jun 22, 2010 at 9:29 AM, Karl Denninger <> wrote:
Justin Graf wrote:
On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote: 
Would moving WAL dir to separate disk help potentially ?     
Yes it can have a big impact.
WAL on a separate spindle will make a HUGE difference in performance.  TPS rates frequently double OR BETTER with WAL on a dedicated spindle.

Strongly recommended.

Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL is on a different physical disk.  Snapshotting the data disk where WAL is on a separate spindle and backing it up **WILL NOT WORK** and **WILL** result in an non-restoreable backup.

The manual discusses this but it's easy to miss.... don't or you'll get a NASTY surprise if something goes wrong.....

-- Karl


--
Sent via pgsql-performance mailing list ()
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


От:
Scott Carey
Дата:

On Jun 22, 2010, at 7:29 AM, Karl Denninger wrote:

> Justin Graf wrote:
>> 
>> On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote:
>>   
>>> Would moving WAL dir to separate disk help potentially ?
>>>    
>>>     
>> 
>> Yes it can have a big impact.
> WAL on a separate spindle will make a HUGE difference in performance.  TPS rates frequently double OR BETTER with WAL
ona dedicated spindle.
 
> 
> Strongly recommended.
> 

Most of the performance increase on Linux, if your RAID card has a data-safe write-back cache (battery or solid state
cachepersistence in case of power failure), is a separate file system, not separate spindle.   Especially if ext3 is
usedwith the default "ordered" journal, it is absolutely caustic to performance to have WAL and data on the same file
system. 
 

The whole 'separate spindle' thing is important if you don't have a good caching raid card, otherwise it doesn't matter
thatmuch.
 


> Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL is on a different physical disk.
Snapshottingthe data disk where WAL is on a separate spindle and backing it up **WILL NOT WORK** and **WILL** result in
annon-restoreable backup.
 
> 
> The manual discusses this but it's easy to miss.... don't or you'll get a NASTY surprise if something goes
wrong.....
> 
> -- Karl
> <karl.vcf><ATT00001..txt>


От:
Ivan Voras
Дата:

On 06/22/10 16:40, Greg Smith wrote:
> Grzegorz Jaśkiewicz wrote:
>> raid: serveRAID M5014 SAS/SATA controller
>>
>
> Do the "performant servers" have a different RAID card?  This one has
> terrible performance, and could alone be the source of your issue.  The
> ServeRAID cards are slow in general, and certainly slow running RAID10.

What are some good RAID10 cards nowadays?

On the other hand, RAID10 is simple enough that soft-RAID
implementations should be more than adequate - any ideas why a dedicated
card has it "slow"?

От:
Florian Weimer
Дата:

* Ivan Voras:

> On the other hand, RAID10 is simple enough that soft-RAID
> implementations should be more than adequate - any ideas why a dedicated
> card has it "slow"?

Barrier support on RAID10 seems to require some smallish amount of
non-volatile storage which supports a high number of write operations
per second, so a software-only solution might not be available.

--
Florian Weimer                <>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99

От:
Ivan Voras
Дата:

On 06/23/10 14:00, Florian Weimer wrote:
> * Ivan Voras:
>
>> On the other hand, RAID10 is simple enough that soft-RAID
>> implementations should be more than adequate - any ideas why a dedicated
>> card has it "slow"?
>
> Barrier support on RAID10 seems to require some smallish amount of
> non-volatile storage which supports a high number of write operations
> per second, so a software-only solution might not be available.

If I understand you correctly, this can be said in general for all
spinning-disk usage and is not specific to RAID10. (And in the case of
high, constant TPS, no amount of NVRAM will help you).

От:
Matthew Wakeling
Дата:

On Wed, 23 Jun 2010, Ivan Voras wrote:
> On 06/23/10 14:00, Florian Weimer wrote:
>> Barrier support on RAID10 seems to require some smallish amount of
>> non-volatile storage which supports a high number of write operations
>> per second, so a software-only solution might not be available.
>
> If I understand you correctly, this can be said in general for all
> spinning-disk usage and is not specific to RAID10. (And in the case of
> high, constant TPS, no amount of NVRAM will help you).

No. Write barriers work fine with a single disc, assuming it is set up
correctly. The barrier is a command telling the disc to make sure that one
piece of data is safe before starting to write another piece of data.

However, as soon as you have multiple discs, the individual discs do not
have a way of communicating with each other to make sure that the first
piece of data is written before the other. That's why you need a little
bit of non-volatile storage to mediate that to properly support barriers.

Of course, from a performance point of view, yes, you need some NVRAM on
any kind of spinning storage to maintain high commit rates.

Matthew

--
 I wouldn't be so paranoid if you weren't all out to get me!!

От:
Scott Marlowe
Дата:

On Wed, Jun 23, 2010 at 8:25 AM, Ivan Voras <> wrote:
> On 06/23/10 14:00, Florian Weimer wrote:
>> * Ivan Voras:
>>
>>> On the other hand, RAID10 is simple enough that soft-RAID
>>> implementations should be more than adequate - any ideas why a dedicated
>>> card has it "slow"?
>>
>> Barrier support on RAID10 seems to require some smallish amount of
>> non-volatile storage which supports a high number of write operations
>> per second, so a software-only solution might not be available.
>
> If I understand you correctly, this can be said in general for all
> spinning-disk usage and is not specific to RAID10. (And in the case of
> high, constant TPS, no amount of NVRAM will help you).

Not entirely true.  Let's say you have enough battery backed cache to
hold 10,000 transaction writes in memory at once.  The RAID controller
can now re-order those writes so that they go from one side of the
disk to the other, instead of randomly all over the place.  That will
most certainly help improve your throughput.

От:
Scott Marlowe
Дата:

On Wed, Jun 23, 2010 at 6:06 AM, Ivan Voras <> wrote:
> On 06/22/10 16:40, Greg Smith wrote:
>> Grzegorz Jaśkiewicz wrote:
>>> raid: serveRAID M5014 SAS/SATA controller
>>>
>>
>> Do the "performant servers" have a different RAID card?  This one has
>> terrible performance, and could alone be the source of your issue.  The
>> ServeRAID cards are slow in general, and certainly slow running RAID10.
>
> What are some good RAID10 cards nowadays?

LSI, Areca, 3Ware (now LSI I believe)

> On the other hand, RAID10 is simple enough that soft-RAID
> implementations should be more than adequate - any ideas why a dedicated
> card has it "slow"?

This is mostly a problem with some older cards that focused on RAID-5
performance, and RAID-10 was an afterthought.  On many of these cards
(older PERCs for instance) it was faster to either use a bunch of
RAID-1 pairs in hardware with RAID-0 in software on top, or put the
thing into JBOD mode and do it all in software.