Re: Investigating IO Saturation

Brad Nicholson wrote:
> I'm investigating a potential IO issue.  We're running 7.4 on AIX
> 5.1.  During periods of high activity (reads, writes, and vacuums), we
> are seeing iostat reporting 100% disk usage.  I have a feeling that
> the iostat numbers are misleading.  I can make iostat usage jump from
> less than 10% to greater than 95% by running a single vacuum against a
> moderate sized table (no noticeable change in the other activity).
>
Well that isn't surprising. Vacuum is brutal especially on 7.4 as that
is pre background writer. What type of IO do you have available (RAID,
SCSI?)

> Do I actually have a problem with IO?  Whether I do or not, at what
> point should I start to be concerned about IO problems?  If my
> understanding is correct, it should be based on the wait time.  Here's
> the output of vmstat during heavy load (reads, writes, several daily
> vacuums and a nightly pg_dump).  Wait times appear to be alright, but
> my understanding of when to start being concerned about IO starvation
> is foggy at best.
>
> vmstat 5
> kthr     memory             page              faults        cpu
> ----- ----------- ------------------------ ------------ -----------
> r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa
> 2  2 1548418 67130   0   0   0 141   99   0 1295 22784 13128 11  4 71 14
> 3  3 1548422 66754   0   0   0 2127 2965   0 2836 29981 25091 26  4 39 31
> 2  3 1548423 66908   0   0   0 2369 3221   0 3130 34725 28424 25  7 38 30
> 3  5 1548423 67029   0   0   0 2223 3097   0 2722 31885 25929 26  9 33 32
> 3  3 1548423 67066   0   0   0 2366 3194   0 2824 43546 36226 30  5 35 31
> 2  4 1548423 67004   0   0   0 2123 3236   0 2662 25756 21841 22  4 39 35
> 2  4 1548957 66277   0   0   0 1928 10322   0 2941 36340 29906 28  6
> 34 33
> 3  5 1549245 66024   0   0   0 2324 14291   0 2872 39413 25615 25  4
> 34 37
> 2  6 1549282 66107   0   0   0 1930 11189   0 2832 72062 32311 26  5
> 32 38
> 2  4 1549526 65855   0   0   0 2375 9278   0 2822 40368 32156 29  5 37 29
> 2  3 1548984 66227   0   0   0 1732 5065   0 2825 39240 30788 26  5 40 30
> 3  4 1549341 66027   0   0   0 2325 6453   0 2790 37567 30509 28  5 37 30
> 2  4 1549377 65789   0   0   0 1633 2731   0 2648 35533 27395 20  5 39 36
> 1  5 1549765 65666   0   0   0 2272 3340   0 2792 43002 34090 26  5 29 40
> 2  3 1549787 65646   0   0   0 1779 2679   0 2596 37446 29184 22  5 37 36
> 2  5 1548985 66263   0   0   0 2077 3086   0 2778 49579 39940 26  9 35 30
> 2  4 1548985 66473   0   0   0 2078 3093   0 2682 23274 18460 22  3 41 34
> 4  3 1548985 66263   0   0   0 2177 3344   0 2734 43029 35536 29  5 38 28
> 1  4 1548985 66491   0   0   0 1978 3215   0 2739 28291 22672 23  4 41 32
> 3  3 1548985 66422   0   0   0 1732 2469   0 2852 71865 30850 28  5 38 29
>