Обсуждение: software or hardware RAID?

We aren't sure whether to use software MDRaid or a MegaRAID card.

We're buying some new Postgres servers with 

    2 x 240GB Intel SSD S4610 (RAID1 : system)
    4 x 960GB Intel SSD S4610 (RAID10 : db)

We'll be using Postgres 11 on Debian.

The MegaRAID 9271-8i with flash cache protection is available from our
provider. I think they may also have the 9361-8i which is 12Gb/s.

Our current servers which use the LSI 9261 with SSDs and we don't see
any IO significant load as we are in RAM most of the time and the RAID
card seems to flatten out any IO spikes.

We use MDRaid elsewhere but we've never used it for our databases
before.

Advice gratefully received.

Rory

[A previous version of this was sent to the PostgreSQL performance list]
 

On 17/03/19, Rory Campbell-Lange (rory@campbell-lange.net) wrote:
> We aren't sure whether to use software MDRaid or a MegaRAID card.
> 
> We're buying some new Postgres servers with 
> 
>     2 x 240GB Intel SSD S4610 (RAID1 : system)
>     4 x 960GB Intel SSD S4610 (RAID10 : db)
> 
> We'll be using Postgres 11 on Debian.
> 
> The MegaRAID 9271-8i with flash cache protection is available from our
> provider. I think they may also have the 9361-8i which is 12Gb/s.
> 
> Our current servers which use the LSI 9261 with SSDs and we don't see
> any IO significant load as we are in RAM most of the time and the RAID
> card seems to flatten out any IO spikes.
> 
> We use MDRaid elsewhere but we've never used it for our databases
> before.

Apologies for re-heating this email from last week. I could really do with the
advice.

Has anyone got any general comments on whether software RAID or an LSI card
is preferable?

We will be replicating load on an existing server, which has an LSI 9261 card.
Below is some stats from sar showing a "heavy" period of load on vdisk sda

    00:00:01  DEV     tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
    14:15:01  sda  112.82    643.09  14986.24    138.53      2.09     18.50      0.25      2.86
    14:25:01  sda  108.52    270.17  15682.94    147.01      1.87     17.22      0.25      2.73
    14:35:01  sda  107.96    178.25  14868.52    139.37      1.70     15.73      0.23      2.53
    14:45:01  sda  150.97    748.94  16919.69    117.03      1.83     12.11      0.22      3.28

Thanks for any advice.

Rory 

On 3/23/19 7:09 AM, Rory Campbell-Lange wrote:
> On 17/03/19, Rory Campbell-Lange (rory@campbell-lange.net) wrote:
>> We aren't sure whether to use software MDRaid or a MegaRAID card.
>>
>> We're buying some new Postgres servers with
>>
>>      2 x 240GB Intel SSD S4610 (RAID1 : system)
>>      4 x 960GB Intel SSD S4610 (RAID10 : db)
>>
>> We'll be using Postgres 11 on Debian.
>>
>> The MegaRAID 9271-8i with flash cache protection is available from our
>> provider. I think they may also have the 9361-8i which is 12Gb/s.
>>
>> Our current servers which use the LSI 9261 with SSDs and we don't see
>> any IO significant load as we are in RAM most of the time and the RAID
>> card seems to flatten out any IO spikes.
>>
>> We use MDRaid elsewhere but we've never used it for our databases
>> before.
> 
> Apologies for re-heating this email from last week. I could really do with the
> advice.
> 
> Has anyone got any general comments on whether software RAID or an LSI card
> is preferable?
> 
> We will be replicating load on an existing server, which has an LSI 9261 card.
> Below is some stats from sar showing a "heavy" period of load on vdisk sda
> 
>     00:00:01  DEV     tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
>     14:15:01  sda  112.82    643.09  14986.24    138.53      2.09     18.50      0.25      2.86
>     14:25:01  sda  108.52    270.17  15682.94    147.01      1.87     17.22      0.25      2.73
>     14:35:01  sda  107.96    178.25  14868.52    139.37      1.70     15.73      0.23      2.53
>     14:45:01  sda  150.97    748.94  16919.69    117.03      1.83     12.11      0.22      3.28
> 
> Thanks for any advice.
> 
> Rory
> 


I have run both software and hardware (though different than the card you listed), and had good success with both.  In
caseswhere I had little money, just drop 6 drives into a md raid 10, and run happy for years and years.  I run
productionPG 11 on software raid 10 as we speek.
 

I personally prefer software raid, for a few reasons:
1) you'll probably be running on a batter backup anyway, so missing raid card battery isn't that much
2) 100% compatible with any other hardware you wanna run.  Sucky thing about hardware card is your on that one
forever.
3) tooling is much better and simpler.  I really hate the crappy bios raid screen.  I never know if adding an HD to an
exitingraid will wipe it or maintain it.
 
4) I setup smartctl to watch and report on drives.  Even a 50% chance it detects before failure is a net benefit.  You
cantalways to that through hardware raid
 

You can always start with software raid, see how it runs for a while, then buy hardware raid if its not working out.

-Andy

On 23/03/19, Andy Colson (andy@squeakycode.net) wrote:
> On 3/23/19 7:09 AM, Rory Campbell-Lange wrote:
> > On 17/03/19, Rory Campbell-Lange (rory@campbell-lange.net) wrote:
...
> > > We're buying some new Postgres servers with
> > > 
> > >      2 x 240GB Intel SSD S4610 (RAID1 : system)
> > >      4 x 960GB Intel SSD S4610 (RAID10 : db)
> > > 
> > > We'll be using Postgres 11 on Debian.
> > > 
> > > The MegaRAID 9271-8i with flash cache protection is available from our
> > > provider. I think they may also have the 9361-8i which is 12Gb/s.
> > > 
> > > Our current servers which use the LSI 9261 with SSDs and we don't see
> > > any IO significant load as we are in RAM most of the time and the RAID
> > > card seems to flatten out any IO spikes.
> > > 
> > > We use MDRaid elsewhere but we've never used it for our databases
> > > before.

> > Has anyone got any general comments on whether software RAID or an LSI card
> > is preferable?
> > 
> > We will be replicating load on an existing server, which has an LSI 9261 card.
> > Below is some stats from sar showing a "heavy" period of load on vdisk sda
> > 
> >     00:00:01  DEV     tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
> >     14:15:01  sda  112.82    643.09  14986.24    138.53      2.09     18.50      0.25      2.86
> >     14:25:01  sda  108.52    270.17  15682.94    147.01      1.87     17.22      0.25      2.73
> >     14:35:01  sda  107.96    178.25  14868.52    139.37      1.70     15.73      0.23      2.53
> >     14:45:01  sda  150.97    748.94  16919.69    117.03      1.83     12.11      0.22      3.28

> I have run both software and hardware (though different than the card you listed), and had good success with both.
Incases where I had little money, just drop 6 drives into a md raid 10, and run happy for years and years.  I run
productionPG 11 on software raid 10 as we speek.
 
> 
> I personally prefer software raid, for a few reasons:
> 1) you'll probably be running on a batter backup anyway, so missing raid card battery isn't that much
> 2) 100% compatible with any other hardware you wanna run.  Sucky thing about hardware card is your on that one
forever.
> 3) tooling is much better and simpler.  I really hate the crappy bios raid screen.  I never know if adding an HD to
anexiting raid will wipe it or maintain it.
 
> 4) I setup smartctl to watch and report on drives.  Even a 50% chance it detects before failure is a net benefit.
Youcant always to that through hardware raid
 
> 
> You can always start with software raid, see how it runs for a while, then buy hardware raid if its not working out.

Thanks very much for the comments, Andy.

If money was no object, would you choose a fancy hardware RAID card?

You are right that the SSDs we are purchasing have enough cache + power to not need a BBU, and I agree that the
managementtools for software raid are much more convenient.
 

Rory

On 3/23/19 11:51 AM, Rory Campbell-Lange wrote:
> On 23/03/19, Andy Colson (andy@squeakycode.net) wrote:
>> On 3/23/19 7:09 AM, Rory Campbell-Lange wrote:
>>> On 17/03/19, Rory Campbell-Lange (rory@campbell-lange.net) wrote:
> ...
>>>> We're buying some new Postgres servers with
>>>>
>>>>       2 x 240GB Intel SSD S4610 (RAID1 : system)
>>>>       4 x 960GB Intel SSD S4610 (RAID10 : db)
>>>>
>>>> We'll be using Postgres 11 on Debian.
>>>>
>>>> The MegaRAID 9271-8i with flash cache protection is available from our
>>>> provider. I think they may also have the 9361-8i which is 12Gb/s.
>>>>
>>>> Our current servers which use the LSI 9261 with SSDs and we don't see
>>>> any IO significant load as we are in RAM most of the time and the RAID
>>>> card seems to flatten out any IO spikes.
>>>>
>>>> We use MDRaid elsewhere but we've never used it for our databases
>>>> before.
> 
>>> Has anyone got any general comments on whether software RAID or an LSI card
>>> is preferable?
>>>
>>> We will be replicating load on an existing server, which has an LSI 9261 card.
>>> Below is some stats from sar showing a "heavy" period of load on vdisk sda
>>>
>>>     00:00:01  DEV     tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
>>>     14:15:01  sda  112.82    643.09  14986.24    138.53      2.09     18.50      0.25      2.86
>>>     14:25:01  sda  108.52    270.17  15682.94    147.01      1.87     17.22      0.25      2.73
>>>     14:35:01  sda  107.96    178.25  14868.52    139.37      1.70     15.73      0.23      2.53
>>>     14:45:01  sda  150.97    748.94  16919.69    117.03      1.83     12.11      0.22      3.28
> 
>> I have run both software and hardware (though different than the card you listed), and had good success with both.
Incases where I had little money, just drop 6 drives into a md raid 10, and run happy for years and years.  I run
productionPG 11 on software raid 10 as we speek.
 
>>
>> I personally prefer software raid, for a few reasons:
>> 1) you'll probably be running on a batter backup anyway, so missing raid card battery isn't that much
>> 2) 100% compatible with any other hardware you wanna run.  Sucky thing about hardware card is your on that one
forever.
>> 3) tooling is much better and simpler.  I really hate the crappy bios raid screen.  I never know if adding an HD to
anexiting raid will wipe it or maintain it.
 
>> 4) I setup smartctl to watch and report on drives.  Even a 50% chance it detects before failure is a net benefit.
Youcant always to that through hardware raid
 
>>
>> You can always start with software raid, see how it runs for a while, then buy hardware raid if its not working
out.
> 
> Thanks very much for the comments, Andy.
> 
> If money was no object, would you choose a fancy hardware RAID card?
> 
> You are right that the SSDs we are purchasing have enough cache + power to not need a BBU, and I agree that the
managementtools for software raid are much more convenient.
 
> 
> Rory
> 


That's a tough question.  I don't even have SSD's yet.  I can't say what performance is like on HW Raid SSD vs SW Raid
SSD. If money where no object, and I wanted performance over all else, I'd benchmark both.
 

If usability were >= performance, I'd go with extra ram, and SW Raid SSD.  And a BMW. :-)

-Andy

On 23/03/19, Hans Schou (hans.schou@gmail.com) wrote:
> On Sun, Mar 17, 2019 at 11:54 PM Rory Campbell-Lange <
> rory@campbell-lange.net> wrote:
> 
> > We aren't sure whether to use software MDRaid or a MegaRAID card.
> 
> Never go with hardRaid.  I have had a breakdown on a hardware RAID and as
> it was special and not off-the-shelf, I could not move the disk to another
> controller. I think it was a capacitor, maybe capasitor plaegue.
> Only thing I had to do was to restore to the day before and the customer
> lost one days work.
> 
> From that on, I only use softRAID.

Thanks for that advice, Hans.

On Sat, Mar 23, 2019 at 12:09:11PM +0000, Rory Campbell-Lange wrote:
> On 17/03/19, Rory Campbell-Lange (rory@campbell-lange.net) wrote:
> > We aren't sure whether to use software MDRaid or a MegaRAID card.
> > 
> > We're buying some new Postgres servers with 
> > 
> >     2 x 240GB Intel SSD S4610 (RAID1 : system)
> >     4 x 960GB Intel SSD S4610 (RAID10 : db)
> > 
> > We'll be using Postgres 11 on Debian.
> > 
> > The MegaRAID 9271-8i with flash cache protection is available from our
> > provider. I think they may also have the 9361-8i which is 12Gb/s.
> > 
> > Our current servers which use the LSI 9261 with SSDs and we don't see
> > any IO significant load as we are in RAM most of the time and the RAID
> > card seems to flatten out any IO spikes.
> > 
> > We use MDRaid elsewhere but we've never used it for our databases
> > before.
> 
> Apologies for re-heating this email from last week. I could really do with the
> advice.
> 
> Has anyone got any general comments on whether software RAID or an LSI card
> is preferable?
> 
> We will be replicating load on an existing server, which has an LSI 9261 card.
> Below is some stats from sar showing a "heavy" period of load on vdisk sda
> 
>     00:00:01  DEV     tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
>     14:15:01  sda  112.82    643.09  14986.24    138.53      2.09     18.50      0.25      2.86
>     14:25:01  sda  108.52    270.17  15682.94    147.01      1.87     17.22      0.25      2.73
>     14:35:01  sda  107.96    178.25  14868.52    139.37      1.70     15.73      0.23      2.53
>     14:45:01  sda  150.97    748.94  16919.69    117.03      1.83     12.11      0.22      3.28
> 
> Thanks for any advice.
> Rory 

Hi Rory,

The main reason, in my opinion, to use a HW RAID card is for the NVRAM
battery backed cache to support writing to traditional spinning disks.
Since your SSDs have power-loss support, you do not need that and the HW
RAID controller. For database use, you would almost certainly be using
RAID 10 and software RAID 10 is extremely performant. I am in the middle
of setting up a new system with NVMe SSD drives and HW RAID would be a
terrible bottle-neck and software RAID is really the only realistice
option.

Regards,
Ken

On 28/03/19, Perumal Raj (perucinci@gmail.com) wrote:
> On Sat, Mar 23, 2019 at 3:12 PM Kenneth Marshall <ktm@rice.edu> wrote:
> > On Sat, Mar 23, 2019 at 12:09:11PM +0000, Rory Campbell-Lange wrote:
> > > On 17/03/19, Rory Campbell-Lange (rory@campbell-lange.net) wrote:

> > > > We aren't sure whether to use software MDRaid or a MegaRAID card.
...
> > > > We use MDRaid elsewhere but we've never used it for our databases
> > > > before.
> > >
> > > Apologies for re-heating this email from last week. I could really
> > > do with the advice.
> > >
> > > Has anyone got any general comments on whether software RAID or an
> > > LSI card is preferable?

...

> > The main reason, in my opinion, to use a HW RAID card is for the NVRAM
> > battery backed cache to support writing to traditional spinning disks.
> > Since your SSDs have power-loss support, you do not need that and the HW
> > RAID controller. For database use, you would almost certainly be using
> > RAID 10 and software RAID 10 is extremely performant. I am in the middle
> > of setting up a new system with NVMe SSD drives and HW RAID would be a
> > terrible bottle-neck and software RAID is really the only realistice
> > option.

> Just would like to know conclusion here ,
> 
> What is best RAID method (Software Or Hardware) for Postgres DB and what
> level ?

Hi Perumal

Ken's comments summarise the general tenor of the advice I've been
given.

Rory