Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

On 02.11.2010 16:40, Heikki Linnakangas wrote:
> On 02.11.2010 16:30, Tom Lane wrote:
>> Heikki Linnakangas writes:
>>> I think we can fix this by requiring that any multi-WAL-record actions
>>> that are in-progress when a checkpoint starts (at the REDO-pointer) must
>>> finish before the checkpoint record is written.
>>
>> What happens if someone wants to start a new split while the checkpoint
>> is hanging fire?
>
> You mean after CreateCheckPoint has determined the redo pointer, but
> before it has written the checkpoint record? The new split can go ahead,
> and the checkpoint doesn't need care about it. Recovery will start at
> the redo pointer, so it will see the split record, and will know to
> finish the incomplete split if necessary.
>
> The logic is the same as with inCommit. Checkpoint will fetch the list
> of in-progress splits some time after determining the redo-pointer. It
> will then wait until all of those splits have finished. Any new splits
> that begin after fetching the list don't affect the checkpoint.
>
> inCommit can't be used as is, because it's tied to the Xid, but
> something similar should work.

Here's a first draft of this, using the inCommit flag as is. It works, 
but suffers from starvation if you have a lot of concurrent 
multi-WAL-record actions. I tested that by running INSERTs to a table 
with tsvector field with a GiST index on it from five concurrent 
sessions, and saw checkpoints regularly busy-waiting for over a minute.

To avoid that, we need something a little bit more complicated than a 
boolean flag. I'm thinking of adding a counter beside the inCommit flag 
that's incremented every time a new multi-WAL-record action begins, so 
that the checkpoint process can distinguish between a new action that 
was started after deciding the REDO pointer and an old one that's still 
running.

(inCommit is a misnomer now, of course. Will need to find a better name..)

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com

Re: B-tree parent pointer and checkpoints

On 08.11.2010 15:40, Heikki Linnakangas wrote:
> Here's a first draft of this, using the inCommit flag as is. It works,
> but suffers from starvation if you have a lot of concurrent
> multi-WAL-record actions. I tested that by running INSERTs to a table
> with tsvector field with a GiST index on it from five concurrent
> sessions, and saw checkpoints regularly busy-waiting for over a minute.
>
> To avoid that, we need something a little bit more complicated than a
> boolean flag. I'm thinking of adding a counter beside the inCommit flag
> that's incremented every time a new multi-WAL-record action begins, so
> that the checkpoint process can distinguish between a new action that
> was started after deciding the REDO pointer and an old one that's still
> running.
>
> (inCommit is a misnomer now, of course. Will need to find a better name..)

Here's a 2nd version, with an additional counter in PGPROC to avoid 
starving checkpoint in the face of a constant stream e.g GiST inserts.

The new rule is that before you start a multi-WAL-record operation that 
needs to be completed at end of recovery if you crash in the middle, you 
call HoldCheckpoint(), and once you're finished, ResumeCheckpoint(). 
rm_safe_restartpoint() is gone.

This is a pre-existing bug, but given the lack of field reports and the 
fact that it's pretty darn hard to run into this in real life, I'm 
inclined to not backpatch this.

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Re: B-tree parent pointer and checkpoints

Intelligent RDBMS