> > Yes, I am able to reproduce this, basically, until we get the last > tuple of the multi insert we can not clear the toast data otherwise we > can never form a complete tuple. So the only possible fix I can think > of is to consider the multi-insert WAL without the final multi-insert > tuple as partial data then we will avoid streaming until we get the > complete WAL of one multi-insert. I am working on the patch to fix > this, I will share that in some time.
The attached patch should fix the issue, now the output is like below
Thanks. This looks fine to me. We should still be able to stream multi-insert transactions (COPY) as and when the copy buffer becomes full and is flushed. That seems to be a reasonable restriction to me.
We should incorporate the regression test in the final patch. I am not entirely sure if what I have done is acceptable (or even works in all scenarios). We could possibly have a long list of tuples instead of doing the exponential magic. Or we should consider lowering the min value for logical_decoding_work_mem and run these tests with a much lower value. In fact, that's how I caught the problem in the first place. I had deliberately lowered the value to 1kB so that streaming code kicks in very often and even for small transactions.