Re: CDC/ETL system on top of logical replication with pgoutput, custom client

Поиск
Список
Период
Сортировка
От Ashutosh Bapat
Тема Re: CDC/ETL system on top of logical replication with pgoutput, custom client
Дата
Msg-id CAExHW5ttk51J5taF6vJUF5sSrgf=TxSF134tA4Q5+KMwLZ=PXQ@mail.gmail.com
обсуждение исходный текст
Ответ на RE: CDC/ETL system on top of logical replication with pgoutput, custom client  (José Neves <rafaneves3@msn.com>)
Ответы RE: CDC/ETL system on top of logical replication with pgoutput, custom client  (José Neves <rafaneves3@msn.com>)
Список pgsql-hackers
On Tue, Oct 24, 2023 at 8:53 PM José Neves <rafaneves3@msn.com> wrote:
>
> Hi there, hope to find you well.
>
> I have a follow-up question to this already long thread.
>
> Upon deploying my PostgreSQL logical replication fed application on a stale database, I ended up running out of
space,as the replication slot is being held back till the next time that we receive a data-changing event, and we
advanceto that new LSN offset. 
> I think that the solution for this is to advance our LSN offset every time a keep-alive message is received ('k' //
107).
> My doubt is, can the keep-alive messages be received in between open transaction events? I think not, but I would
liketo get your input to be extra sure as if this happens, and I commit that offset, I may introduce again faulty logic
leadingto data loss. 
>
> In sum, something like this wouldn't happen:
> BEGIN LSN001
> INSERT LSN002
> KEEP LIVE LSN003
> UPDATE LSN004
> COMMIT LSN005
>

If the downstream acknowledges receipt of LSN003 and saves it locally
and crashes, upon restart the upstream will resend all the
transactions that committed after LSN003 including the one ended at
LSN005. So this is safe.

--
Best Wishes,
Ashutosh



В списке pgsql-hackers по дате отправления:

Предыдущее
От: Smolkin Grigory
Дата:
Сообщение: race condition in pg_class
Следующее
От: José Neves
Дата:
Сообщение: RE: CDC/ETL system on top of logical replication with pgoutput, custom client