The problem is that transactions are delivered to replica through single channel: logical replication slot. And while such transaction is waiting acknowledgement from arbiter, it is blocking replication channel preventing other (parallel transactions) from been replicated and applied.
Streaming interleaved xacts from the slot as discussed in the prior mail would help there.
You'd continue to apply concurrent work from other xacts, and just handle commit messages as they arrive, sending the confirmations back through the DTM API.
I have implemented pool of background workers. May be it will be useful not only for me.
Excellent.
It should be possible to make that a separate extension. You can use C functions from other extensions by exposing a single pg_proc function with 'internal' return type that populates a struct of function pointers for the API. A single DirectFunctionCall lets you get the API struct. That's how pglogical_output handles hooks. The main downside is that you can't do that without a connection to a database with the extension installed so the pg_proc entry is exposed.
So it could make more sense to just keep it as a separate .c / .h file that's copied into trees that use it. Simpler and easier, but uglier.
It consists of one produces-multiple consumers queue implemented using buffer in shared memory, spinlock and two semaphores.
[snip]
You just place in this queue some bulk of bytes (work, size), it is placed in queue and then first available worker will dequeue it and execute.
Very nice.
To handle xact streaming I think you're likely to need a worker dispatch key too, where the dispatch keys are "sticky" to a given worker. So you assign xid 1231 to a worker at BEGIN. Send all work to the pool and everything with xid 1231 goes to that worker. At commit you clear the assignment of xis 1231.
Alternately a variant of the Execute method that lets you dispatch to a specific worker would do the job.