But looks like there is not so much sense in having multiple network connection between one pair of nodes. It seems to be better to have one connection between nodes, but provide parallel execution of received transactions at destination side. But it seems to be also nontrivial. We have now in PostgreSQL some infrastructure for background works, but there is still no abstraction of workers pool and job queue which can provide simple way to organize parallel execution of some jobs. I wonder if somebody is working now on it or we should try to propose our solution?
There are definitely two clear places where additional help would be useful and welcome right now.
1. Allowing logical decoding to have a "speculative pre-commit data" option, to allow some data to be made available via the decoding api, allowing data to be transferred prior to commit.
Something relevant I ran into re this:
in reorderbuffer.c, on ReorderBufferCommit:
* We currently can only decode a transaction's contents in when their commit
* record is read because that's currently the only place where we know about
* cache invalidations. Thus, once a toplevel commit is read, we iterate over
* the top and subtransactions (using a k-way merge) and replay the changes in
* lsn order.
I haven't dug into the implications particularly as I'm chasing something else, but want to note it on the thread. Here be dragons when it comes to transaction streaming.