Re: GDQ iimplementation
От | Marko Kreen |
---|---|
Тема | Re: GDQ iimplementation |
Дата | |
Msg-id | AANLkTin53YIQSfuqSQHzuTN1uoyoHbQKSI7yRud8UC4f@mail.gmail.com обсуждение исходный текст |
Ответ на | Re: GDQ iimplementation (Jan Wieck <JanWieck@Yahoo.com>) |
Список | pgsql-cluster-hackers |
On 5/11/10, Jan Wieck <JanWieck@yahoo.com> wrote: > On 5/11/2010 11:20 AM, Marko Kreen wrote: > > On 5/11/10, Jan Wieck <JanWieck@yahoo.com> wrote: > > > On 5/11/2010 9:36 AM, Marko Kreen wrote: > > > > And although PgQ can operate with any N >= 2 segments, it queries > > > > on 2 at a time, same as Slony. Rest are just there to give admins > > > > some safety room for "OH F*CK" moments. With short rotation times, > > > > it starts to seem useful.. > > > > > > > > > > Agreed. The rotation time should actually reflect the longest running > > > transactions experienced on a frequent base from the application. And > there > > > needs to be a safeguard against rotating over even longer running > > > transactions. > > > > > > > Nightly pg_dump.. ;) > > > > > > > The problem with a long running transaction is that it could have > written > > > into log segment 1 before we switched to segment 2. We can only TRUNCATE > > > segment 1 after that transaction committed AND the log has been consumed > by > > > everyone interested in it. > > > > > > I am not familiar with how PgQ/Londiste do this. Slony specifically > > > remembers the highest XID in progress at the time of switching, waits > until > > > the lowest XID in progress is higher than that (so all log that ever > went > > > into that segment is now visible or aborted), then waits for all log in > that > > > segment to be confirmed and finally truncates the log. All this time, it > > > needs to do the UNION query over both log segments. > > > > > > > The "highest XID" means actually "own transaction" here? > > And it's not committed yet? That's seems to leave transactions > > that happen before it's own commit into dubious state? > > > > One needs to tell transactions to switch log, commit, then look at the > highest running XID after that. Any XID lower/equal to that one could > possibly have written into the old segment. Yeah, sounds fine. Except you cannot ignore the newer table with that. But that makes difference only for consumers that are lagging. > > Although you may be fine, if you don't try to minimize > > reading both tables. > > > > PgQ does this: > > > > Rotate: > > 1) If some consumer reads older table, don't rotate. > > 2) Set table_nr++, switch_step1 = txid_current(), switch_step2 = NULL > > 3) Commit > > 4) Set switch_step2 = txid_current() where switch_step2 IS NULL > > 5) Commit > > > > Right, exactly like that :) > > > > Reader: > > 1) xmin1 = xmin of lower snapshot of batch > > 2) xmax2 = xmax of higher snapshot of batch > > 3) if xmax2 < switch_step1, read older table > > 4) if xmin1 > switch_step2, read newer table > > 5) otherwise read both > > > > Sounds familiar. I still don't know exactly what role the 3rd log segment > plays in that, but it sure cannot hurt. It makes sure you have one rotation_period of events always available. In case you want to do some recovery on them. But that's it. -- marko
В списке pgsql-cluster-hackers по дате отправления: