Обсуждение: Missing empty transaction optimization in pgoutput plugin
Hi All, test_decoding output plugin skips sending stream_start and stream_end if no change from a segment of a streamed transaction is sent downstream. If no change has been sent downstream across all the streamed segments it doesn't send commit or abort for that streamed transaction. But I don't see similar changes in stream related callbacks in pgoutput output plugin. While the changes discussed in thread [1] were committed, those didn't include this optimization for test_decoding plugin as well. The optimization was discussed in [2] separately as a bug fix. Maybe because it wasn't considered as an optimization that time, we may not have considered it for pgoutput plugin which doesn't have explicit skip-empty-xacts option. I think the optimization would be useful in setups where multiple publications and long transactions are common. Any reason not to have it. [1] https://www.postgresql.org/message-id/flat/688b0b7f-2f6c-d827-c27b-216a8e3ea700%402ndquadrant.com [2] https://www.postgresql.org/message-id/flat/CAA4eK1%2BOqgFNZkf7%3DETe_y5ntjgDk3T0wcdkd4Sot_u1hySGfw%40mail.gmail.com -- Best Wishes, Ashutosh Bapat
On Mon, Dec 8, 2025 at 12:19 PM Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote: > > test_decoding output plugin skips sending stream_start and stream_end > if no change from a segment of a streamed transaction is sent > downstream. If no change has been sent downstream across all the > streamed segments it doesn't send commit or abort for that streamed > transaction. But I don't see similar changes in stream related > callbacks in pgoutput output plugin. > > While the changes discussed in thread [1] were committed, those didn't > include this optimization for test_decoding plugin as well. The > optimization was discussed in [2] separately as a bug fix. Maybe > because it wasn't considered as an optimization that time, we may not > have considered it for pgoutput plugin which doesn't have explicit > skip-empty-xacts option. I think the optimization would be useful in > setups where multiple publications and long transactions are common. > Any reason not to have it. > I don't remember any reason why we can't have the same optimization for streamed transactions. IIRC, we implemented this optimization because we got reports for extra messages for short empty-transactions but if you see any practical cases where it could be useful for streamed transactions as well then feel free to propose a patch. -- With Regards, Amit Kapila.
On Monday, December 8, 2025 2:49 PM Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote: > > Hi All, > test_decoding output plugin skips sending stream_start and stream_end if no > change from a segment of a streamed transaction is sent downstream. If no > change has been sent downstream across all the streamed segments it > doesn't send commit or abort for that streamed transaction. But I don't see > similar changes in stream related callbacks in pgoutput output plugin. > > While the changes discussed in thread [1] were committed, those didn't > include this optimization for test_decoding plugin as well. The optimization > was discussed in [2] separately as a bug fix. Maybe because it wasn't > considered as an optimization that time, we may not have considered it for > pgoutput plugin which doesn't have explicit skip-empty-xacts option. I think > the optimization would be useful in setups where multiple publications and > long transactions are common. > Any reason not to have it. One possible reason we did not implement the optimization to avoid sending empty stream blocks along with commit d5a9d86 is the handling of prepared transactions, for which STREAM PREPARE is sent. The logical replication worker currently relies on receiving at least one STREAM START/STREAM STOP prior to this; otherwise, it might encounter errors when applying STREAM PREPARE. Although the optimization is feasible, it requires some modifications to ensure the replication worker works correctly, such as sending an empty stream block before issuing STREAM PREPARE. One might think that we could avoid sending STREAM PREPARE for empty txn as well, but that could be tricky, see the comment atop of (typedef struct PGOutputTxnData) for details. > > [1] https://www.postgresql.org/message-id/flat/688b0b7f-2f6c-d827-c27b- > 216a8e3ea700%402ndquadrant.com > [2] https://www.postgresql.org/message- > id/flat/CAA4eK1%2BOqgFNZkf7%3DETe_y5ntjgDk3T0wcdkd4Sot_u1hySGfw% > 40mail.gmail.com Best Regards, Hou zj
On Wed, Dec 10, 2025 at 9:17 AM Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> wrote: > > On Monday, December 8, 2025 2:49 PM Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> wrote: > > > > Hi All, > > test_decoding output plugin skips sending stream_start and stream_end if no > > change from a segment of a streamed transaction is sent downstream. If no > > change has been sent downstream across all the streamed segments it > > doesn't send commit or abort for that streamed transaction. But I don't see > > similar changes in stream related callbacks in pgoutput output plugin. > > > > While the changes discussed in thread [1] were committed, those didn't > > include this optimization for test_decoding plugin as well. The optimization > > was discussed in [2] separately as a bug fix. Maybe because it wasn't > > considered as an optimization that time, we may not have considered it for > > pgoutput plugin which doesn't have explicit skip-empty-xacts option. I think > > the optimization would be useful in setups where multiple publications and > > long transactions are common. > > Any reason not to have it. > > One possible reason we did not implement the optimization to avoid sending empty stream > blocks along with commit d5a9d86 is the handling of prepared transactions, for > which STREAM PREPARE is sent. The logical replication worker currently relies on > receiving at least one STREAM START/STREAM STOP prior to this; otherwise, it > might encounter errors when applying STREAM PREPARE. Although the optimization > is feasible, it requires some modifications to ensure the replication worker works > correctly, such as sending an empty stream block before issuing STREAM PREPARE. > > One might think that we could avoid sending STREAM PREPARE for empty txn as > well, but that could be tricky, see the comment atop of (typedef struct > PGOutputTxnData) for details. > > > > > [1] https://www.postgresql.org/message-id/flat/688b0b7f-2f6c-d827-c27b- > > 216a8e3ea700%402ndquadrant.com > > [2] https://www.postgresql.org/message- > > id/flat/CAA4eK1%2BOqgFNZkf7%3DETe_y5ntjgDk3T0wcdkd4Sot_u1hySGfw% > > 40mail.gmail.com > Thanks a lot for the details. I see it's much larger change than just a few lines like test_decoding.c. -- Best Wishes, Ashutosh Bapat