Here's our hardware break down.
The logvg on the new hardware is 30MB/s slower (170 MB/s vs 200 MB/s ) than the logvg on the older hardware which was an immediately interesting difference but we have yet to be able to create a test scenario that successfully implicates this slower log speed in our problems. That is something we are actively working on.
Old server hardware:
Manufacturer: Dell Inc.
Product Name: PowerEdge R810
4x Intel(R) Xeon(R) CPU E7540 @ 2.00GHz
32x16384 MB 1066 MHz DDR3
Controller 0: PERC H700 - 2 disk RAID-1 278.88 GB rootvg
Controller 1: PERC H800 - 18 disk RAID-6 2,178.00 GB datavg, 4 drive RAID-10 272.25 GB logvg, 2 hot spare
2x 278.88 GB 15K SAS on controller 0
24x 136.13 GB 15K SAS on controller 1
New server hardware:
Manufacturer: Dell Inc.
Product Name: PowerEdge R820
4x Intel(R) Xeon(R) CPU E5-4620 0 @ 2.20GHz
32x32 GB 1333 MHz DDR3
Controller 0: PERC H710P - 4 disk RAID-6 557.75 GB rootvg
Controller 1: PERC H810 - 20 disk RAID-60 4,462.00 GB datavg, 2 disk RAID-1 278.88 GB logvg, 2 hot spare
28x278.88 GB 15K SAS drives total.