Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements
От | Matthias van de Meent |
---|---|
Тема | Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements |
Дата | |
Msg-id | CAEze2WgHFnYdxkNUmvqxOc-cFUNEYaTqL7+Pei=CtA-ZrTOFyw@mail.gmail.com обсуждение исходный текст |
Ответ на | Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements (Michail Nikolaev <michail.nikolaev@gmail.com>) |
Ответы |
Re: Revisiting {CREATE INDEX, REINDEX} CONCURRENTLY improvements
(Michail Nikolaev <michail.nikolaev@gmail.com>)
|
Список | pgsql-hackers |
On Thu, 1 Feb 2024, 17:06 Michail Nikolaev, <michail.nikolaev@gmail.com> wrote: > > > > > I just realised there is one issue with this design: We can't cheaply > > > > reset the snapshot during the second table scan: > > > > It is critically important that the second scan of R/CIC uses an index > > > > contents summary (made with index_bulk_delete) that was created while > > > > the current snapshot was already registered. I think the best way for this to work would be an index method that exclusively stores TIDs, and of which we can quickly determine new tuples, too. I was thinking about something like GIN's format, but using (generation number, tid) instead of ([colno, colvalue], tid) as key data for the internal trees, and would be unlogged (because the data wouldn't have to survive a crash). Then we could do something like this for the second table scan phase: 0. index->indisready is set [...] 1. Empty the "changelog index", resetting storage and the generation number. 2. Take index contents snapshot of new index, store this. 3. Loop until completion: 4a. Take visibility snapshot 4b. Update generation number of the changelog index, store this. 4c. Take index snapshot of "changelog index" for data up to the current stored generation number. Not including, because we only need to scan that part of the index that were added before we created our visibility snapshot, i.e. TIDs labeled with generation numbers between the previous iteration's generation number (incl.) and this iteration's generation (excl.). 4d. Combine the current index snapshot with that of the "changelog" index, and save this. Note that this needs to take care to remove duplicates. 4e. Scan segment of table (using the combined index snapshot) until we need to update our visibility snapshot or have scanned the whole table. This should give similar, if not the same, behavour as that which we have when we RIC a table with several small indexes, without requiring us to scan a full index of data several times. Attemp on proving this approach's correctness: In phase 3, after each step 4b: All matching tuples of the table that are in the visibility snapshot: * Were created before scan 1's snapshot, thus in the new index's snapshot, or * Were created after scan 1's snapshot but before index->indisready, thus not in the new index's snapshot, nor in the changelog index, or * Were created after the index was set as indisready, and committed before the previous iteration's visibility snapshot, thus in the combined index snapshot, or * Were created after the index was set as indisready, after the previous visibility snapshot was taken, but before the current visibility snapshot was taken, and thus definitely included in the changelog index. Because we hold a snapshot, no data in the table that we should see is removed, so we don't have a chance of broken HOT chains. Kind regards, Matthias van de Meent Neon (https://neon.tech)
В списке pgsql-hackers по дате отправления:
Следующее
От: Tomas VondraДата:
Сообщение: Re: Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE