We want to export data from PG to Kafka,
We can't rely on extension which we have written as there could be any problems which we are not aware of and PG might break.
We don't want our master to go down because of the extension we have written.
So, we are okay with having a new PG instance whose work is just to export data, as slave.
What we thought of doing is to pause recovery (start-up) process, on any vacuum changes on system catalog tables and resume once, our logical-decoding has caught up.
That way, we aren't bloating our master.
Our problem is, we aren't able to exactly identify what WAL records are causing Vacuum changes, as far as our understanding goes, `HEAP_2 (rmgr), CLEAN` and `BTREE (rmgr), VACUUM` records.
Inorder to see our understanding is right or not, IPC (inter-process communication) between WAL_SENDER and START_UP process, is not efficient enough.
I have seen Latches, but I'm not sure exactly how to use them, as from comments, my understanding is START_UP process is not available in PGPROC array.
For efficient inter-process communication, what would be ideal way to communicate between two processes.
I'm new to PostgreSQL, and C - world.
What we are trying to achieve is something similar to this:
START_UP process goes to sleep, as soon as it sees any vacuum on catalog.
WAL_SENDER process will resume recovery (wakeup), as soon as it caught up and goes to sleep.
START_UP process will wake up when THERE is something for WAL_SENDER to send.
Basically, IPC, between two processes, where one process generates work, and other consumes it.
producer should go to sleep, until consumer caught up.
consumer should signal producer of it's completion and wake up producer.
cycle goes on...
As I have indicated before, I new to C and PostgreSQL, I familiar with GoLang, and I have written a sample program in Go (attached below).
Any suggestions and pointers would be greatly helpful.