This is a proposal for load distributed checkpoint.
(It is presented on postgresql anniversary summit in last summer.)
We offen encounters performance gap during checkpoint. The reason is write
bursts. Storage devices are too overworked in checkpoint, so they can not
supply usual transaction processing.
Checkpoint consists of the following four steps, and the major performance
problem is 2nd step. All dirty buffers are written without interval in it.
1. Query information (REDO pointer, next XID etc.)2. Write dirty pages in buffer pool3. Flush all modified files4.
Updatecontrol file
I suggested to write pages with sleeping in 2nd step, using normal activity
of the background writer. It is something like cost-based vacuum delay.
Background writer has two pointers, 'ALL' and 'LRU', indicating where to
write out in buffer pool. We can wait for the ALL clock-hand going around
to guarantee all pages to be written.
Here is pseudo-code for the proposed method. The internal loop is just the
same as bgwriter's activity.
PrepareCheckPoint(); -- do step 1 Reset num_of_scanned_pages by ALL activity; do { BgBufferSync(); -- do a part
ofstep 2 sleep(bgwriter_delay); } while (num_of_scanned_pages < shared_buffers); CreateCheckPoint(); -- do step 3
and4
We may accelerate background writer to reduce works at checkpoint instead of
the method, but it introduces another performance problem; Extra pressure
is always put on the storage devices to keep the number of dirty pages low.
I'm working about adjusting the progress of checkpoint to checkpoint timeout
and wal segments limitation automatically to avoid overlap of two checkpoints.
I'll post a patch sometime soon.
Comments and suggestions welcome.
Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center