Re: XFS options and benchmark woes

> -----Original Message-----
> From: mark [mailto:mark@sm-a.net]
> Sent: Monday, August 08, 2011 12:15 AM
> To: 'pgsql-performance@postgresql.org'
> Subject: XFS options and benchmark woes
>
> Hello PG perf junkies,
>
>
> Sorry this may get a little long winded. Apologies if the formatting
> gets trashed.
>
>
>
> Background:
>
> I have been setting up some new servers for PG and I am getting some
> odd numbers with zcav, I am hoping a second set of eyes here can point
> me in the right direction. (other tests like bonniee++ (1.03e) and dd
> also give me odd (flat and low) numbers)
>
> I will preface this with, yes I bought greg's book. Yes I read it, and
> it has helped me in the past, but seem to have hit an oddity.
>
> (hardware,os, and config stuff listed at the end)
>
>
>
>
>
> Short version: my zcav and dd tests look to get I/O bound. My numbers
> in ZCAV are flat like and SSD which is odd for 15K rpm disks.


uggg, ZCAV numbers appear to be CPU bound. Not i/o .

>
>
>
>
> Long version:
>
>
> In the past when dealing with storage I typically see a large gain with
> moving from ext3 to XFS, provided I set readahead to 16384 on either
> filesystem.
>
> I also see typical down ward trends in the MB/s (expected) and upward
> trends in access times (expected) with either file system.
>
>
> These blades + storage-blades are giving me atypical results .
>
>
> I am not seeing a dramatic down turn in MB/s in zcav nor am I seeing
> access time really increase. (something I have only seen before when I
> forget to have readahead set high enough) things are just flat at about
> 420MB/s in zcav @ .6ms for access time with XFS and ~470MB/s @.56ms for
> ext3.
>
> FWIW I get worthless results with zcav and bonnie++ using 1.03 or 1.96
> sometimes, which isn't something I have had happen before even though
> greg does mention it.
>
>
> Also when running zcav I will see kswapdX (0 and 1 in my two socket
> case) start to eat significant cpu time (~40-50% each), with dd -
> kswapd and pdflush become very active as well. This only happens once
> free mem gets low. As well zcav or dd looks to get CPU bound at 100%
> while i/o wait stays almost at 0.0 most of the time. (iostat -x -d
> shows util % at 98% though). I see this with either XFS or ext3. Also
> when I cat /proc/zoneinfo it looks like I am getting heavy contention
> for a single page in DMA while the tests are running. (see end of email
> for zoneinfo)
>
> Bonnie is giving me 99% cpu usage reported. Watching it while running
> it bounces between 100 and 99. Kswap goes nuts here as well.
>
>
> I am lead to believe that I may need a 2.6.32 (rhel 6.1) or higher
> kernel to see some of the kswapd issues go away. (testing that
> hopefully later this week). Maybe that will take care of everything. I
> don't know yet.
>
>  Side note: Setting vm.swappiness to 10 (or 0) doesn't help, although
> others on the RHEL support site indicated it did fix kswap issues for
> them.
>
>
>
> Running zcav on my home system (4 disk raid 1+0 3ware controller +BBWC
> using ext4 ubunut 2.6.38-8 I don't see zcav near 100% and I see lots of
> i/o wait as expected, and my zoneinfo for DMA doesn't sit at 1)
>
> Not going to focus too much on ext3 since I am pretty sure I should be
> able to get better numbers from XFS.
>
>
>
> With mkfs.xfs I have done some reading and it appears that it can't
> automatically read the stripsize (aka stripe size to anyone other than
> HP) or the number of disks. So I have been using the following:
>
> mkfs.xfs -b size=4k -d su=256k,sw=6,agcount=256
>
> (256K is the default hp stripsize for raid1+0, I have 12 disks in raid
> 10 so I used sw=6, agcount of 256 because that is a (random) number I
> got from google that seemed in the ball park.)
>
>
>
>
>
>
> which gives me:
> meta-data=/dev/cciss/c0d0        isize=256    agcount=256,
> agsize=839936 blks
>          =                       sectsz=512   attr=2
> data     =                       bsize=4096   blocks=215012774,
> imaxpct=25
>          =                       sunit=64     swidth=384 blks
> naming   =version 2              bsize=4096   ascii-ci=0
> log      =internal log           bsize=4096   blocks=32768, version=2
>          =                       sectsz=512   sunit=64 blks, lazy-
> count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
>
>
> (if I don't specif the agcount or su,sw stuff I get
> meta-data=/dev/cciss/c0d0        isize=256    agcount=4,
> agsize=53753194 blks
>          =                       sectsz=512   attr=2
> data     =                       bsize=4096   blocks=215012774,
> imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0
> log      =internal log           bsize=4096   blocks=32768, version=2
>          =                       sectsz=512   sunit=0 blks, lazy-
> count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0)
>
> )
>
>
>
>
>
> So it seems like I should be giving it the extra parameters at mkfs.xfs
> time... could someone confirm ? In the past I have never specified the
> su or sw or ag groups I have taken the defaults. But since I am getting
> odd numbers here I started playing with em. Getting little or no
> change.
>
>
>
> for mounting:
> logbufs=8,noatime,nodiratime,nobarrier,inode64,allocsize=16m
>
>
>
> (I know that noatime also means nodiratime according xfs.org, but in
> the past I seem to get better numbers when having both)
>
> I am using nobarrier because I have a battery backed raid cache and the
> FAQ @ XFS.org seems to indicate that is the right choice.
>
>
> FWIW, if I put sunit and swidth in the mount options it seems to change
> them lower (when viewed with xfs_info) so I haven't been putting it in
> the mount options.
>
>
>
>
> verify readahead:
> blockdev --getra /dev/cciss/c0d0
> 16384
>
>
>
>
>
>
>
> If anyone wants the benchmark outputs I can send them, but basically
> zcav being FLAT for bother MB/s and access time tells me something is
> wrong. And it will take days for me to re run all the ones I have done.
> I didn't save much once I saw results that don't fit with what I
> thought I should get.
>
>
>
>
> I haven't done much with pgbench yet as I figure its pointless to move
> on while the raw I/O numbers look off to me. At that time I am going to
> make the call between wal on the OS raid 1 or going to 10 data disks
> and 2 os and 2 wal.
>
>
>
>
>
>
> I have gone up to 2.6.18-27(something, wanna say 2 or 4) to see if the
> issue went away, it didn't. I have gone back to 2.6.18-238.5 and put in
> a new CCISS driver directly from HP, and the issue also does not go
> away. People at work are thinking it might kernel bug that we have
> somehow never notice before which is why we are going to look at RHEL
> 6.1.  we tried a 5.3 kernel that someone on rh bugzilla said didn't
> have the issue but this blade had a fit with it - no network, lots of
> other stuff not working and then it kernel panic'd so we quickly gave
> up on that...
>
>
>
> We may try and shoehorn in the 6.1 kernel and a few dependencies as
> well. Moving to RHEL 6.1. will mean a long test period before it can go
> into prod and we want to get this new hardware in sooner than that can
> be done.  (even with all it's problems its probably still faster than
> what it is replacing just from the 48GB of ram and 3 gen newer CPUS)
>
>
>
>
>
>
>
>
> Hardware and config stuff as it sits right now.
>
>
>
> Blade Hardware:
> ProLiant BL460c G7 (bios power flag set to high performance)
> 2 intel 5660 cpus. (HT left on)
> 48GB of ram (12x4GB @ 1333MHz)
> Smart Array P410i (Embedded)
> Points of interest from hpacucli -
>     - Hardware Revision: Rev C
>     - Firmware Version: 3.66
>     - Cache Board Present: True
>     - Elevator Sort: Enabled
>     - Cache Status: OK
>     - Cache Backup Power Source: Capacitors
>        - Battery/Capacitor Count: 1
>        - Battery/Capacitor Status: OK
>     - Total Cache Size: 512 MB
>        - Accelerator Ratio: 25% Read / 75% Write
>     - Strip Size: 256 KB
>     - 2x 15K RPM 146GB 6Gbps SAS in raid 1 for OS (ext3)
>     - Array Accelerator: Enabled
>     - Status: OK
>     - drives firmware = HPD5
>
> Blade Storage subsystem:
> HP SB2200 (12 disk 15K )
>
> Points of interest from hpacucli
>
> Smart Array P410i in Slot 3
>    Controller Status: OK
>    Hardware Revision: Rev C
>    Firmware Version: 3.66
>    Elevator Sort: Enabled
>    Wait for Cache Room: Disabled
>    Cache Board Present: True
>    Cache Status: OK
>    Accelerator Ratio: 25% Read / 75% Write
>    Drive Write Cache: Disabled
>    Total Cache Size: 1024 MB
>    No-Battery Write Cache: Disabled
>    Cache Backup Power Source: Capacitors
>    Battery/Capacitor Count: 1
>    Battery/Capacitor Status: OK
>    SATA NCQ Supported: True
>
>
>       Logical Drive: 1
>          Size: 820.2 GB
>          Fault Tolerance: RAID 1+0
>          Heads: 255
>          Sectors Per Track: 32
>          Cylinders: 65535
>          Strip Size: 256 KB
>          Status: OK
>          Array Accelerator: Enabled
>          Disk Name: /dev/cciss/c0d0
>          Mount Points: /raid 820.2 GB
>          OS Status: LOCKED
>
> 12 drives in Raid 1+0, using XFS.
>
>
> OS:
> OS: RHEL 5.6 (2.6.18-238.9.1.el5)
> Database use: PG 9.0.2 for OLTP.
>
>
> CCISS info:
> filename:       /lib/modules/2.6.18-
> 238.9.1.el5/kernel/drivers/block/cciss.ko
> version:        3.6.22-RH1
> description:    Driver for HP Controller SA5xxx SA6xxx version 3.6.22-
> RH1
> author:         Hewlett-Packard Company
>
> XFS INFO:
> xfsdump-2.2.48-3.el5
> xfsprogs-2.10.2-7.el5
>
> head of ZONEINFO while zcav is running and kswap is going nuts:
> the min,low,high of 1 seems odd to me. On other systems these get above
> 1.
>
> Node 0, zone      DMA
>   pages free     2493
>         min      1
>         low      1
>         high     1
>         active   0
>         inactive 0
>         scanned  0 (a: 3 i: 3)
>         spanned  4096
>         present  2393
>     nr_anon_pages 0
>     nr_mapped    1
>     nr_file_pages 0
>     nr_slab      0
>     nr_page_table_pages 0
>     nr_dirty     0
>     nr_writeback 0
>     nr_unstable  0
>     nr_bounce    0
>     numa_hit     0
>     numa_miss    0
>     numa_foreign 0
>     numa_interleave 0
>     numa_local   0
>     numa_other   0
>         protection: (0, 3822, 24211, 24211)
>   pagesets
>   all_unreclaimable: 1
>   prev_priority:     12
>   start_pfn:         0
>
>
>
> numastat (probably worthless since I have been pounding on this box for
> a while before capturing it)
>
>                            node0           node1
> numa_hit              3126413031       247696913
> numa_miss               95489353      2781917287
> numa_foreign          2781917287        95489353
> interleave_hit             81178           97872
> local_node            3126297257       247706110
> other_node              95605127      2781908090
>
>
>