On 13/07/2010 3:53 PM, Andras Fabian wrote:
> Hi Scott,
>
> No, we didn't have a kernel update (it is still the stock Ubuntu 10.04 Server kernel ... 2.6.32.2). And in the
meantime- this morning - I have discovered, that the rebooted server is again slowing down! It is not at the level of
thenot-rebooted-server (about 45 mins for the 3 Gig file)... it "only" needs 22 minutes, but it is already quite a bit
awayfrom the optimum of 3 minutes (or less). So, definitely, something is "deteriorating" in the system ...
>
> And I also did dome readings with iostat -xd 5 ... And the target drive to which the output of the STDOUT is directed
isbelow 1% utilization (mostly around 0.2 - 0.4%) with rare "peaks around 2-3% when it does a little bit more. And this
ismaybe one of the interesting observations. It seems to periodically "flush" a bit more out, just to fall asleep again
(withminimum write activity). The drive, from which the reads come (the one, where PG-s data files are ... it is the
8-diskRAID 10), has a little bit more activity (utilization 6-8%) but this data is also - concurrently - in use by some
appsreading from the DB (just, normal traffic on the DB).
I don't think it'll be particularly helpful in this case, but you never
know, so: another information-collecting tool is "blktrace". This can
let you observe in detail exactly what's being done on a given block
device. It's helpful when you have weird I/O patterns and can't figure
out why, as it'll often reveal some process continually poking away at
some file it doesn't need to, thrashing away on a disk-backed mmap()ed
tempfile, or the like.
--
Craig Ringer