Ответ: WAL and indexes (Re: [HACKERS] WAL status & todo)

>I don't understand why WAL needs to log internal operations of any of
>the index types.  Seems to me that you could treat indexes as black
>boxes that are updated as side effects of WAL log items for heap tuples:
>when adding a heap tuple as a result of a WAL item, you just call the
>usual index insert routines, and when deleting a heap tuple as a result

On recovery backend *can't* use any usual routines:
system catalogs are not available.

>of undoing a WAL item, you mark the tuple invalid but don't physically
>remove it till VACUUM (thus no need to worry about its index entries).

One of the purposes of WAL is immediate removing tuples
inserted by aborted xactions. I want make VACUUM
*optional* in future - space must be available for
reusing without VACUUM. And this is first, very small,
step in this direction.

>This doesn't address the issue of recovering from an incomplete index
>update (such as a partially-completed btree page split), but I think
>the most reliable way to do that is to add WAL records on the order of
>"update beginning for index X" and "update done for index X".  If you
>see the begin and not the done record when replaying a log, you assume

You will still have to log changes for *each* page
updated on behalf of index operation! The fact that
you've seen begin/end records in log doesn't mean
that all intermediate changes to index pages are
written to index file unless you've logged all these
changes and see all of them in index on recovery.

>the index is corrupt and rebuild it from scratch, using Hiroshi's
>index-rebuild code.

How fast is rebuilding of index for table with
10^7 records?
I agree to consider rtree/hash/gist as experimental
index access methods BUT we have to have at least
*one* reliable index AM with short down time/
fast recovery.

>For that matter I am far from convinced that the currently committed
>code for btree WAL logging is correct --- where does it cope with
>cleaning up after an unfinished page split?  I don't see it.

What do you mean? If you say about updating parent
page ("my bits moved ..." etc) then as I've mentioned
previously we can handle uninserted parent item in
run time (though it's not implemented yet -:)).
WAL allows to restore both left and right siblings
and this is the most critical split issue.
(BTW, note that having all btitems on leaf level
at place we could do REINDEX veeeeery fast).

>Since we have very poor testing capabilities for the non-mainstream
>index types (remember how I broke rtree completely during 6.5 devel,
>and no one noticed till quite late in beta?) I will have absolutely
>zero confidence in WAL support for these index types if it's implemented
>this way.  I think we should go with a black-box approach that's the
>same for all index types and is implemented completely outside the
>index-access-method-specific code.

I agreed with this approach for all indices except
btree (above + "hey, something is already done for
them" -:)). But remember that to know is index
consistent or not we have to log *all* changes made
in index file anyway... so seems we have to be
very close to be AM specific -:)

Vadim