Re: checkpointer continuous flushing

>>>>   flsh |      full speed tps       | percent of late tx, 4 clients
>>>>   /srt |  1 client   |  4 clients  |   100 |   200 |   400 |
>>>>    N/N | 173 +- 289* | 198 +- 531* | 27.61 | 43.92 | 61.16 |
>>>>    N/Y | 458 +- 327* | 743 +- 920* |  7.05 | 14.24 | 24.07 |
>>>>    Y/N | 169 +- 166* | 187 +- 302* |  4.01 | 39.84 | 65.70 |
>>>>    Y/Y | 546 +- 143  | 681 +- 459  |  1.55 |  3.51 |  2.84 |
>>>>
>>>> The effect of sorting is very positive (+150% to 270% tps). On this run,
>>>>
>>> flushing has a positive (+20% with 1 client) or negative (-8 % with 4
>>> clients) on throughput, and late transactions are reduced by 92-95% when
>>> both options are activated.
>>>
>>> Why there is dip in performance with multiple clients,
>>
>> I'm not sure to see the "dip". The performances are better with 4 clients
>> compared to 1 client?
>
> What do you mean by "negative (-8 % with 4 clients) on throughput" in 
> above sentence? I thought by that you mean that there is dip in TPS with 
> patch as compare to HEAD at 4 clients.

Ok, I misunderstood your question. I thought you meant a dip between 1 
client and 4 clients. I meant that when flush is turned on tps goes down 
by 8% (743 to 681 tps) on this particular run. Basically tps improvements 
mostly come from "sort", and "flush" has uncertain effects on tps 
(throuput), but much more on latency and performance stability (lower late 
rate, lower standard deviation).

Note that I'm not comparing to HEAD in the above tests, but with the new 
options desactivated, which should be more or less comparable to current 
HEAD, i.e. there is no sorting nor flushing done, but this is not strictly 
speaking HEAD behavior. Probably I should get some figures with HEAD as 
well to check the "more or less" assumption.

> Also I am not completely sure what's +- means in your data above?

The first figure before "+-" is the tps, the second after is its standard 
deviation computed in per-second traces. Some runs are very bad, with 
pgbench stuck at times, and result on stddev larger than the average, they 
ere noted with "*".

> I understand your point and I also don't have any specific answer
> for it at this moment, the point of worry is that it should not lead
> to degradation of certain cases as compare to current algorithm.
> The workload where it could effect is when your data doesn't fit
> in shared buffers, but can fit in RAM.

Hmmm. My point of view is still that the logical priority is to optimize 
for disk IO first, then look for compatible RAM optimisations later.

I can run tests with a small shared_buffers, but probably it would just 
trigger a lot of checkpoints, or worse rely on the bgwriter to find space, 
which would generate random IOs.

-- 
Fabien.