Обсуждение: Effects of setting linux block device readahead size

Hi all,

I've started to display the effects of changing the Linux block device
readahead buffer to the sequential read performance using fio.  There
are lots of raw data buried in the page, but this is what I've
distilled thus far.  Please have a look and let me know what you
think:

http://wiki.postgresql.org/wiki/HP_ProLiant_DL380_G5_Tuning_Guide#Readahead_Buffer_Size

Regards,
Mark

On Tue, 9 Sep 2008, Mark Wong wrote:

> I've started to display the effects of changing the Linux block device
> readahead buffer to the sequential read performance using fio.

Ah ha, told you that was your missing tunable.  I'd really like to see the
whole table of one disk numbers re-run when you get a chance.  The
reversed ratio there on ext2 (59MB read/92MB write) was what tipped me off
that something wasn't quite right initially, and until that's fixed it's
hard to analyze the rest.

Based on your initial data, I'd say that the two useful read-ahead
settings for this system are 1024KB (conservative but a big improvement)
and 8192KB (point of diminishing returns).  The one-disk table you've got
(labeled with what the default read-ahead is) and new tables at those two
values would really flesh out what each disk is capable of.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

On Wed, Sep 10, 2008 at 9:26 AM, Scott Carey <scott@richrelevance.com> wrote:
> How does that readahead tunable affect random reads or mixed random /
> sequential situations?  In many databases, the worst case scenarios aren't
> when you have a bunch of concurrent sequential scans but when there is
> enough random read/write concurrently to slow the whole thing down to a
> crawl.   How the file system behaves under this sort of concurrency
>
> I would be very interested in a mixed fio profile with a "background writer"
> doing moderate, paced random and sequential writes combined with concurrent
> sequential reads and random reads.

The data for the other fio profiles we've been using are on the wiki,
if your eyes can take the strain.  We are working on presenting the
data in a more easily digestible manner.  I don't think we'll add any
more fio profiles in the interest of moving on to doing some sizing
exercises with the dbt2 oltp workload.  We're just going to wrap up a
couple more scenarios first and get through a couple of conference
presentations.  The two conferences in particular are the Linux
Plumbers Conference, and the PostgreSQL Conference: West 08, which are
both in Portland, Oregon.

Regards,
Mark

On Wed, 10 Sep 2008, Scott Carey wrote:

> How does that readahead tunable affect random reads or mixed random /
> sequential situations?

It still helps as long as you don't make the parameter giant.  The read
cache in a typical hard drive noawadays is 8-32MB.  If you're seeking a
lot, you still might as well read the next 1MB or so after the block
requested once you've gone to the trouble of moving the disk somewhere.
Seek-bound workloads will only waste a relatively small amount of the
disk's read cache that way--the slow seek rate itself keeps that from
polluting the buffer cache too fast with those reads--while sequential
ones benefit enormously.

If you look at Mark's tests, you can see approximately where the readahead
is filling the disk's internal buffers, because what happens then is the
sequential read performance improvement levels off.  That looks near 8MB
for the array he's tested, but I'd like to see a single disk to better
feel that out.  Basically, once you know that, you back off from there as
much as you can without killing sequential performance completely and that
point should still support a mixed workload.

Disks are fairly well understood physical components, and if you think in
those terms you can build a gross model easily enough:

Average seek time:      4ms
Seeks/second:        250
Data read/seek:        1MB    (read-ahead number goes here)
Total read bandwidth:    250MB/s

Since that's around what a typical interface can support, that's why I
suggest a 1MB read-ahead shouldn't hurt even seek-only workloads, and it's
pretty close to optimal for sequential as well here (big improvement from
the default Linux RA of 256 blocks=128K).  If you know your work is biased
heavily toward sequential scans, you might pick the 8MB read-ahead
instead.  That value (--setra=16384 -> 8MB) has actually been the standard
"start here" setting 3ware suggests on Linux for a while now:
http://www.3ware.com/kb/Article.aspx?id=11050

> I would be very interested in a mixed fio profile with a "background writer"
> doing moderate, paced random and sequential writes combined with concurrent
> sequential reads and random reads.

Trying to make disk benchmarks really complicated is a path that leads to
a lot of wasted time.  I one made this gigantic design plan for something
that worked like the PostgreSQL buffer management system to work as a disk
benchmarking tool.  I threw it away after confirming I could do better
with carefully scripted pgbench tests.

If you want to benchmark something that looks like a database workload,
benchmark a database workload.  That will always be better than guessing
what such a workload acts like in a synthetic fashion.  The "seeks/second"
number bonnie++ spits out is good enough for most purposes at figuring out
if you've detuned seeks badly.

"pgbench -S" run against a giant database gives results that look a lot
like seeks/second, and if you mix multiple custom -f tests together it
will round-robin between them at random...

It's really helpful to measure these various disk subsystem parameters
individually.  Knowing the sequential read/write, seeks/second, and commit
rate for a disk setup is mainly valuable at making sure you're getting the
full performance expected from what you've got.  Like in this example,
where something was obviously off on the single disk results because reads
were significantly slower than writes.  That's not supposed to happen, so
you know something basic is wrong before you even get into RAID and such.
Beyond confirming whether or not you're getting approximately what you
should be out of the basic hardware, disk benchmarks are much less useful
than application ones.

With all that, I think I just gave away what the next conference paper
I've been working on is about.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

On Wed, 10 Sep 2008, Scott Carey wrote:

> Ok, so this is a drive level parameter that affects the data going into the
> disk cache?  Or does it also get pulled over the SATA/SAS link into the OS
> page cache?

It's at the disk block driver level in Linux, so I believe that's all
going into the OS page cache.  They've been rewriting that section a bit
and I haven't checked it since that change (see below).

> Additionally, I would like to know how this works with hardware RAID -- Does
> it set this value per disk?

Hardware RAID controllers usually have their own read-ahead policies that
may or may not impact whether the OS-level read-ahead is helpful.  Since
Mark's tests are going straight into the RAID controller, that's why it's
helpful here, and why many people don't ever have to adjust this
parameter.  For example, it doesn't give a dramatic gain on my Areca card
even in JBOD mode, because that thing has its own cache to manage with its
own agenda.

Once you start fiddling with RAID stripe sizes as well the complexity
explodes, and next thing you know you're busy moving the partition table
around to make the logical sectors line up with the stripes better and
similar exciting work.

> Additionally, the O/S should have a good heuristic based read-ahead process
> that should make the drive/device level read-ahead much less important.  I
> don't know how long its going to take for Linux to do this right:
> http://archives.postgresql.org/pgsql-performance/2006-04/msg00491.php
> http://kerneltrap.org/node/6642

That was committed in 2.6.23:

http://kernelnewbies.org/Linux_2_6_23#head-102af265937262a7a21766ae58fddc1a29a5d8d7

but clearly some larger minimum hints still helps, as the system we've
been staring at benchmarks has that feature.

> Some chunk of data in that seek is free, afterwords it is surely not...

You can do a basic model of the drive to get a ballpark estimate on these
things like I threw out, but trying to break down every little bit gets
hairy.  In most estimation cases you see, where 128kB is the amount being
read, the actual read time is so small compared to the rest of the numbers
that it just gets ignored.

I was actually being optimistic about how much cache can get filled by
seeks.  If the disk is spinning at 15000RPM, that's 4ms to do a full
rotation.  That means that on average you'll also wait 2ms just to get the
heads lined up to read that one sector on top of the 4ms seek to get in
the area; now we're at 6ms before you've read anything, topping seeks out
at under 167/second.  That number--average seek time plus half a
rotation--is what a lot of people call the IOPS for the drive.  There,
typically the time spent actually reading data once you've gone through
all that doesn't factor in.  IOPS is not very well defined, some people
*do* include the reading time once you're there; one reason I don't like
to use it.  There's a nice chart showing some typical computations here at
http://www.dbasupport.com/oracle/ora10g/disk_IO_02.shtml if anybody wants
to see how this works for other classes of disk.  The other reason I don't
like focusing too much on IOPS (some people act like it's the only
measurement that matters) is that it tells you nothing about the
sequential read rate, and you have to consider both at once to get a clear
picture--particularly when there are adjustments that impact those two
oppositely, like read-ahead.

As far as the internal transfer speed of the heads to the drive's cache
once it's lined up, those are creeping up toward the 200MB/s range for the
kind of faster drives the rest of these stats come from.  So the default
of 128kB is going to take 0.6ms, while a full 1MB might take 5ms.  You're
absolutely right to question how hard that will degrade seek performance;
these slightly more accurate numbers suggest that might be as bad as going
from 6.6ms to 11ms per seek, or from 150 IOPS to 91 IOPS.  It also points
out how outrageously large the really big read-ahead numbers are once
you're seeking instead of sequentially reading.

One thing it's hard to know is how much read-ahead the drive was going to
do on its own, no matter what you told it, anyway as part of its caching
algorithm.

> I suppose I should learn more about pgbench.

Most people use it as just a simple benchmark that includes a mixed
read/update/insert workload.  But that's internally done using a little
command substition "language" that let's you easily write things like
"generate a random number between 1 and 1M, read the record from this
table, and then update this associated record" that scale based on how big
the data set you've given it is.  You an write your own scripts in that
form too.  And if you specify several scripts like that at a time, it will
switch between them at random, and you can analyze the average execution
time broken down per type if you save the latency logs. Makes it real easy
to adjust the number of clients and the mix of things you have them do.

The main problem: it doesn't scale to large numbers of clients very well.
But it can easily simulate 50-100 banging away at a time which is usually
enough to rank filesystem concurrency capabilities, for example.  It's
certainly way easier to throw together a benchmark using it that is
similar to an abstract application than it is to try and model multi-user
database I/O using fio.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Greg Smith wrote:
> Average seek time:      4ms
> Seeks/second:        250
> Data read/seek:        1MB    (read-ahead number goes here)
> Total read bandwidth:    250MB/s
>
Most spinning disks now are nearer to 100MB/s streaming.  You've talked
yourself into twice that, random access!

James

On Wed, Sep 10, 2008 at 11:21 PM, James Mansion
<james@mansionfamily.plus.com> wrote:
> Greg Smith wrote:
>>
>> Average seek time:      4ms
>> Seeks/second:        250
>> Data read/seek:        1MB    (read-ahead number goes here)
>> Total read bandwidth:    250MB/s
>>
> Most spinning disks now are nearer to 100MB/s streaming.  You've talked
> yourself into twice that, random access!

The fastest cheetahs on this page hit 171MB/second:

http://www.seagate.com/www/en-us/products/servers/cheetah/

Are there any drives that have a faster sequential transfer rate out there?

Checked out hitachi's global storage site and they're fastest drive
seems just a tad slower.

On Thu, 11 Sep 2008, James Mansion wrote:

> Most spinning disks now are nearer to 100MB/s streaming.  You've talked
> yourself into twice that, random access!

The point I was trying to make there is that even under impossibly optimal
circumstances, you'd be hard pressed to blow out the disk's read cache
with seek-dominated data even if you read a lot at each seek point.  That
idea didn't make it from my head into writing very well though.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Greg Smith wrote:
> The point I was trying to make there is that even under impossibly
> optimal circumstances, you'd be hard pressed to blow out the disk's
> read cache with seek-dominated data even if you read a lot at each
> seek point.  That idea didn't make it from my head into writing very
> well though.
>
Isn't there a bigger danger in blowing out the cache on the controller
and causing premature pageout of its dirty pages?

If you could get the readahead to work on the drive and not return data
to the controller, that might be dandy, but I'm sceptical.

James

On Thu, 11 Sep 2008, Scott Carey wrote:

> Drives have their own read-ahead in the firmware.  Many can keep track of 2
> or 4 concurrent file accesses.  A few can keep track of more.  This also
> plays in with the NCQ or SCSI command queuing implementation.
>
> Consumer drives will often read-ahead much more than server drives optimized
> for i/o per second.
> The difference in read-ahead sensitivity between the two setups I tested may
> be due to one setup using nearline-SAS (SATA, tuned for io-per sec using SAS
> firmware) and the other used consumer SATA.
> For example, here is one "nearline SAS" style server tuned drive versus a
> consumer tuned one:
>
http://www.storagereview.com/php/benchmark/suite_v4.php?typeID=10&testbedID=4&osID=6&raidconfigID=1&numDrives=1&devID_0=354&devID_1=348&devCnt=2
>
> The Linux readahead setting is _definitely_ in the kernel, definitely uses
> and fills the page cache, and from what I can gather, simply issues extra
> I/O's to the hardware beyond the last one requested by an app in certain
> situations.  It does not make your I/O request larger, it just queues an
> extra I/O following your request.

that extra I/O will be merged with your request by the I/O scheduler code
so that by the time it gets to the drive it will be a single request.

by even if it didn't, most modern drives read the entire cylinder into
their buffer so any additional requests to the drive will be satisfied
from this buffer and not have to wait for the disk itself.

David Lang

> On Thu, Sep 11, 2008 at 12:54 PM, James Mansion <
> james@mansionfamily.plus.com> wrote:
>
>> Greg Smith wrote:
>>
>>> The point I was trying to make there is that even under impossibly optimal
>>> circumstances, you'd be hard pressed to blow out the disk's read cache with
>>> seek-dominated data even if you read a lot at each seek point.  That idea
>>> didn't make it from my head into writing very well though.
>>>
>>>  Isn't there a bigger danger in blowing out the cache on the controller
>> and causing premature pageout of its dirty pages?
>>
>> If you could get the readahead to work on the drive and not return data to
>> the controller, that might be dandy, but I'm sceptical.
>>
>> James
>>
>>
>>
>> --
>> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-performance
>>
>

On Thu, Sep 11, 2008 at 3:36 PM,  <david@lang.hm> wrote:
> On Thu, 11 Sep 2008, Scott Carey wrote:
>
>> Drives have their own read-ahead in the firmware.  Many can keep track of
>> 2
>> or 4 concurrent file accesses.  A few can keep track of more.  This also
>> plays in with the NCQ or SCSI command queuing implementation.
>>
>> Consumer drives will often read-ahead much more than server drives
>> optimized
>> for i/o per second.
>> The difference in read-ahead sensitivity between the two setups I tested
>> may
>> be due to one setup using nearline-SAS (SATA, tuned for io-per sec using
>> SAS
>> firmware) and the other used consumer SATA.
>> For example, here is one "nearline SAS" style server tuned drive versus a
>> consumer tuned one:
>>
>>
http://www.storagereview.com/php/benchmark/suite_v4.php?typeID=10&testbedID=4&osID=6&raidconfigID=1&numDrives=1&devID_0=354&devID_1=348&devCnt=2
>>
>> The Linux readahead setting is _definitely_ in the kernel, definitely uses
>> and fills the page cache, and from what I can gather, simply issues extra
>> I/O's to the hardware beyond the last one requested by an app in certain
>> situations.  It does not make your I/O request larger, it just queues an
>> extra I/O following your request.
>
> that extra I/O will be merged with your request by the I/O scheduler code so
> that by the time it gets to the drive it will be a single request.
>
> by even if it didn't, most modern drives read the entire cylinder into their
> buffer so any additional requests to the drive will be satisfied from this
> buffer and not have to wait for the disk itself.

Generally speaking I agree, but I would still make a separate logical
partition for pg_xlog so that if the OS fills up the /var/log dir or
something, it doesn't impact the db.

On Thu, 11 Sep 2008, Scott Marlowe wrote:

> On Thu, Sep 11, 2008 at 3:36 PM,  <david@lang.hm> wrote:
>> by even if it didn't, most modern drives read the entire cylinder into their
>> buffer so any additional requests to the drive will be satisfied from this
>> buffer and not have to wait for the disk itself.
>
> Generally speaking I agree, but I would still make a separate logical
> partition for pg_xlog so that if the OS fills up the /var/log dir or
> something, it doesn't impact the db.

this is a completely different discussion :-)

while I agree with you in theory, in practice I've seen multiple
partitions cause far more problems than they have prevented (due to the
partitions ending up not being large enough and having to be resized after
they fill up, etc) so I tend to go in the direction of a few large
partitions.

the only reason I do multiple partitions (besides when the hardware or
performance considerations require it) is when I can identify that there
is some data that I would not want to touch on a OS upgrade. I try to make
it so that an OS upgrade can wipe the OS partitions if nessasary.

David Lang

On Thursday 11 September 2008, david@lang.hm wrote:
> while I agree with you in theory, in practice I've seen multiple
> partitions cause far more problems than they have prevented (due to the
> partitions ending up not being large enough and having to be resized
> after they fill up, etc) so I tend to go in the direction of a few large
> partitions.

I used to feel this way until LVM became usable. LVM plus online resizable
filesystems really makes multiple partitions manageable.


--
Alan

On Thu, 11 Sep 2008, Alan Hodgson wrote:

> On Thursday 11 September 2008, david@lang.hm wrote:
>> while I agree with you in theory, in practice I've seen multiple
>> partitions cause far more problems than they have prevented (due to the
>> partitions ending up not being large enough and having to be resized
>> after they fill up, etc) so I tend to go in the direction of a few large
>> partitions.
>
> I used to feel this way until LVM became usable. LVM plus online resizable
> filesystems really makes multiple partitions manageable.

won't the fragmentation of your filesystem across the different LVM
segments hurt you?

David Lang

On Thu, Sep 11, 2008 at 4:33 PM,  <david@lang.hm> wrote:
> On Thu, 11 Sep 2008, Scott Marlowe wrote:
>
>> On Thu, Sep 11, 2008 at 3:36 PM,  <david@lang.hm> wrote:
>>>
>>> by even if it didn't, most modern drives read the entire cylinder into
>>> their
>>> buffer so any additional requests to the drive will be satisfied from
>>> this
>>> buffer and not have to wait for the disk itself.
>>
>> Generally speaking I agree, but I would still make a separate logical
>> partition for pg_xlog so that if the OS fills up the /var/log dir or
>> something, it doesn't impact the db.
>
> this is a completely different discussion :-)
>
> while I agree with you in theory, in practice I've seen multiple partitions
> cause far more problems than they have prevented (due to the partitions
> ending up not being large enough and having to be resized after they fill
> up, etc) so I tend to go in the direction of a few large partitions.

I've never had that problem.  I've always made the big enough.  I
can't imagine building a server where /var/log shared space with my
db.  It's not like every root level dir gets its own partition, but
seriously, logs should never go anywhere that another application is
writing to.

> the only reason I do multiple partitions (besides when the hardware or
> performance considerations require it) is when I can identify that there is
> some data that I would not want to touch on a OS upgrade. I try to make it
> so that an OS upgrade can wipe the OS partitions if nessasary.

it's quite handy to have /home on a separate partition I agree.  But
on most servers /home should be empty.  A few others like /opt or
/usr/local I tend to make a separate one for the reasons you mention
as well.

On Thu, 11 Sep 2008, Alan Hodgson wrote:

> LVM plus online resizable filesystems really makes multiple partitions
> manageable.

I've seen so many reports blaming Linux's LVM for performance issues that
its managability benefits don't seem too compelling.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Scott Carey wrote:
> Consumer drives will often read-ahead much more than server drives
> optimized for i/o per second.
...
> The Linux readahead setting is _definitely_ in the kernel, definitely
> uses and fills the page cache, and from what I can gather, simply
> issues extra I/O's to the hardware beyond the last one requested by an
> app in certain situations.  It does not make your I/O request larger,
> it just queues an extra I/O following your request.
So ... fiddling with settings in Linux is going to force read-ahead, but
the read-ahead data will hit the controller cache and the system buffers.

And the drives use their caches for cyclinder caching implicitly (maybe
the SATA drives appear to preread more because the storage density per
cylinder is higher?)..

But is there any way for an OS or application to (portably) ask SATA,
SAS or SCSI drives to read ahead more (or less) than their default and
NOT return the data to the controller?

I've never heard of such a thing, but I'm no expert in the command sets
for any of this stuff.

James

>
> On Thu, Sep 11, 2008 at 12:54 PM, James Mansion
> <james@mansionfamily.plus.com <mailto:james@mansionfamily.plus.com>>
> wrote:
>
>     Greg Smith wrote:
>
>         The point I was trying to make there is that even under
>         impossibly optimal circumstances, you'd be hard pressed to
>         blow out the disk's read cache with seek-dominated data even
>         if you read a lot at each seek point.  That idea didn't make
>         it from my head into writing very well though.
>
>     Isn't there a bigger danger in blowing out the cache on the
>     controller and causing premature pageout of its dirty pages?
>
>     If you could get the readahead to work on the drive and not return
>     data to the controller, that might be dandy, but I'm sceptical.
>
>     James
>
>
>
>     --
>     Sent via pgsql-performance mailing list
>     (pgsql-performance@postgresql.org
>     <mailto:pgsql-performance@postgresql.org>)
>     To make changes to your subscription:
>     http://www.postgresql.org/mailpref/pgsql-performance
>
>

On Fri, 12 Sep 2008, James Mansion wrote:

> Scott Carey wrote:
>> Consumer drives will often read-ahead much more than server drives
>> optimized for i/o per second.
> ...
>> The Linux readahead setting is _definitely_ in the kernel, definitely uses
>> and fills the page cache, and from what I can gather, simply issues extra
>> I/O's to the hardware beyond the last one requested by an app in certain
>> situations.  It does not make your I/O request larger, it just queues an
>> extra I/O following your request.
> So ... fiddling with settings in Linux is going to force read-ahead, but the
> read-ahead data will hit the controller cache and the system buffers.
>
> And the drives use their caches for cyclinder caching implicitly (maybe the
> SATA drives appear to preread more because the storage density per cylinder
> is higher?)..
>
> But is there any way for an OS or application to (portably) ask SATA, SAS or
> SCSI drives to read ahead more (or less) than their default and NOT return
> the data to the controller?
>
> I've never heard of such a thing, but I'm no expert in the command sets for
> any of this stuff.

I'm pretty sure that's not possible. the OS isn't supposed to even know
the internals of the drive.

David Lang

> James
>
>>
>> On Thu, Sep 11, 2008 at 12:54 PM, James Mansion
>> <james@mansionfamily.plus.com <mailto:james@mansionfamily.plus.com>> wrote:
>>
>>     Greg Smith wrote:
>>
>>         The point I was trying to make there is that even under
>>         impossibly optimal circumstances, you'd be hard pressed to
>>         blow out the disk's read cache with seek-dominated data even
>>         if you read a lot at each seek point.  That idea didn't make
>>         it from my head into writing very well though.
>>
>>     Isn't there a bigger danger in blowing out the cache on the
>>     controller and causing premature pageout of its dirty pages?
>>
>>     If you could get the readahead to work on the drive and not return
>>     data to the controller, that might be dandy, but I'm sceptical.
>>
>>     James
>>
>>
>>
>>     --     Sent via pgsql-performance mailing list
>>     (pgsql-performance@postgresql.org
>>     <mailto:pgsql-performance@postgresql.org>)
>>     To make changes to your subscription:
>>     http://www.postgresql.org/mailpref/pgsql-performance
>>
>>
>
>
>

On Thu, 11 Sep 2008, Scott Carey wrote:
> Preliminary summary:
>
> readahead  |  8 conc read rate  |  1 conc read rate
> 49152  |  311  |  314
> 16384  |  312  |  312
> 12288  |  304  |  309
>  8192  |  292  |
>  4096  |  264  |
>  2048  |  211  |
>  1024  |  162  |  302
>   512  |  108  |
>   256  |  81  | 300
>     8  |  38  |

What io scheduler are you using? The anticipatory scheduler is meant to
prevent this slowdown with multiple concurrent reads.

Matthew


--
And the lexer will say "Oh look, there's a null string. Oooh, there's
another. And another.", and will fall over spectacularly when it realises
there are actually rather a lot.
         - Computer Science Lecturer (edited)