Re: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

Re: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

Masahiko Sawada  writes:
> On Thu, Jul 30, 2026 at 5:52 PM Tom Lane  wrote:
>> The buildfarm looks like this has made 051_effective_wal_level.pl
>> less stable, not more so.  The failures all look like

> What does the buildfarm member produce the above logs? Many buildfarm
> members failed with that error due to commit 6a80179f6b0, but I've not
> seen further failures since pushing commit 18f9785e0b3.

Ah, you are right, there are no failures later than 6aba42c.  I was
misled by the fact that several of the complaining animals are still
red because they've not run since then.  Sorry for the noise.

			regards, tom lane

Re: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

On Thu, Jul 16, 2026 at 6:52 AM Masahiko Sawada  wrote:
>
> On Tue, Jul 14, 2026 at 2:02 PM Masahiko Sawada  wrote:
>
> These races are confined to the narrow window between checking the
> logical decoding status and the new slot becoming visible; once the
> slot is visible, the invalidation performed by the deactivation
> already covers it. So the fix is simple: re-check the logical decoding
> status after the new slot becomes visible.
>

*
- * CheckLogicalDecodingRequirements() must have already errored out if
- * logical decoding is not enabled since we cannot enable the logical
- * decoding status during recovery.
+ * The caller has already checked that logical decoding is enabled via
+ * CheckLogicalDecodingRequirements(), but the status could have been
+ * disabled concurrently before our slot being created: either by
+ * replaying an XLOG_LOGICAL_DECODING_STATUS_CHANGE record, or by
+ * UpdateLogicalDecodingStatusEndOfRecovery() upon promotion. We
+ * cannot enable logical decoding during recovery, so raise an error.
  */
- Assert(IsLogicalDecodingEnabled());
+ if (!IsLogicalDecodingEnabled())
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("logical decoding on standby requires \"effective_wal_level\"
>= \"logical\" on the primary"),
+ errdetail("Logical decoding was concurrently disabled during the
logical replication slot creation.")));

It is not clear after reading the comment above this check what kind
of interlocking would save us from concurrent deactivation by two ways
mentioned by you immediately after this check?

-- 
With Regards,
Amit Kapila.

Re: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

On Thu, Jul 9, 2026 at 9:23 PM Srinath Reddy Sadipiralla
 wrote:
>
>
>
> ---------- Forwarded message ---------
> From: Srinath Reddy Sadipiralla 
> Date: Fri, Jul 10, 2026 at 9:48 AM
> Subject: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation
> To: PostgreSQL Hackers 
> Cc: Masahiko Sawada 
>
>
> Hi,
>
> While stress-testing REPACK CONCURRENTLY, I was in that
> area looking at BUG #19519 (REPACK failing with "missingchunk
> number N for toast value" [0]; note that one reproduces with plain
> REPACK too, since it's the shared copy_table_data() path, so
> it's independent of what follows), I hit a different, reproducible
> failure under high concurrency:
>     ERROR:  unexpected logical decoding status change 1
>
> I believe this is a race bug in the on-demand logical decoding
> activation machinery (logicalctl.c), exposed by
> REPACK CONCURRENTLY but not actually specific to it, as
> this can be reproduced using pg_create_logical_replication_slot and CREATE_REPLICATION_SLOT.
>
> At wal_level = 'replica', logical decoding is toggled on the first time a
> logical slot is created.  EnableLogicalDecoding() sets the shared
> logical_decoding_enabled flag and writes an
> XLOG_LOGICAL_DECODING_STATUS_CHANGE record so standbys
> learn about the change.  On the decoding side, xlog_decode() treats
> that record as unreachable:
> elog(ERROR, "unexpected logical decoding status change %d", ...);
>
> The reasoning (per the comment there) is that no running decoder can ever
> have such a record within the LSN range it scans: there is exactly one
> enable record per disabled->enabled transition, and it precedes the
> decoding start point reserved by any slot.
>
> That case does not hold under concurrency.  EnableLogicalDecoding()
> does:
>
>     LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);
>     if (logical_decoding_enabled)     /* already on? -> done */
>         { ... return; }
>     LWLockRelease(...);
>
>     WaitForProcSignalBarrier(...);    /* lock released here */
>
>     LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);
>     logical_decoding_enabled = true;
>     write_logical_decoding_status_update_record(true);   /* the record */
>     LWLockRelease(...);
>
> The "already enabled?" check and the record write happen under two
> separate lock acquisitions, with the barrier wait in between.  The lock
> must be dropped across the barrier; backends absorbing the barrier
> take LogicalDecodingControlLock in shared mode (via
> IsXLogLogicalInfoEnabled()), so holding it across WaitForProcSignalBarrier()
> would deadlock.
>
> So if two backends create the first logical slot(s) at the same time,
> both can pass the initial check while the flag is still false, and both
> end up writing a status-change record.  The second (redundant) record
> lands after the decoding start point the other slot already reserved,
> so that slot scans it and trips the elog() above.
>
> REPACK CONCURRENTLY makes this easy to hit because each operation
> creates a temporary logical slot (max_repack_replication_slots), so firing
> many REPACKs in parallel from the disabled state gives you many backends
> racing to perform the very first activation.  But nothing here is REPACK-specific.
>
> To reproduce this i have added a test using logical-decoding-activation
> injection point in 051_effective_wal_level.pl
>
> I also hit it with a stress script [1] which was again related to the
> BUG #19519, instead of vacuum full i replaced it with REPACK (concurrently).
>
> To fix this I re-checked the flag after re-acquiring the lock and skipped
> the state change/WAL write if another backend had already completed
> the transition.
>
>     LWLockAcquire(LogicalDecodingControlLock, EXCLUSIVE);
>
>     if (LogicalDecodingCtl->logical_decoding_enabled)
>     {
>         LogicalDecodingCtl->pending_disable = false;
>         LWLockRelease(LogicalDecodingControlLock);
>         return;
>     }
>
>     START_CRIT_SECTION();
>
> With this, only the backend that actually performs the disabled->enabled
> transition writes a record, and every slot's start point stays at or after
> it, restoring the case xlog_decode() depends on.
>
> Patch attached.  It fixes logicalctl.c and adds a regression test to
> 051_effective_wal_level.pl reusing the existing injection-point
> infrastructure.  With the fix the test passes; without it it fails with
> the exact error above.

Thank you for the report and the patch!

I agree with your analysis and the patch basically looks good to me.
Here are some review comments:

+ # Let the released backend finish creating its slot: feed running-xacts
+ # records until it reaches a consistent point (poll_query_until re-runs the
+ # query, so pg_log_standby_snapshot() is called until the slot is created).
+ $primary->poll_query_until(
+ 'postgres', qq[
+select (pg_log_standby_snapshot() is not null)
+  and exists (select 1 from pg_replication_slots
+              where slot_name = 'slot_stray' and confirmed_flush_lsn
is not null)
+]);

I don't think we need to call pg_log_standby_snapshot() until the slot
is created since the slot creation writes the running-xacts record
during the slot creation.

---
+ # Decoding the first slot must not stumble over a stray status-change record.
+ my ($decode_rc, $decode_out, $decode_err) = $primary->psql(
+ 'postgres',
+ qq[select count(*) from pg_logical_slot_get_changes('slot_first',
null, null)],
+ on_error_die => 0);
+ is($decode_rc, 0, "decoding a concurrently-created slot succeeds");
+ unlike(
+ $decode_err,
+ qr/unexpected logical decoding status change/,
+ "no redundant status-change record was decoded");

We can use safe_psql() to check if the query successfully completes.

I've made some cosmetic changes to the comment and the new test
including the above comments. Please review it.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Re: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

On Mon, Jul 20, 2026 at 9:47 AM Srinath Reddy Sadipiralla
 wrote:
>
> Hi Masahiko-san,
>
> On Thu, Jul 16, 2026 at 6:52 AM Masahiko Sawada  wrote:
>>
>>
>> While reviewing concurrency aspects of this code before pushing the
>> fix, I found other race conditions in the same area: logical decoding
>> can be deactivated while a logical slot is being created on a standby.
>>
>> On standbys, logical decoding can be deactivated while a logical slot
>> is being created: either by replaying an
>> XLOG_LOGICAL_DECODING_STATUS_CHANGE record, or by the end-of-recovery
>> transition upon promotion, which deactivates logical decoding if no
>> valid logical slot exists. When logical decoding is deactivated, all
>> logical slots on the standby are invalidated, but this cannot find a
>> slot that is not visible yet. Therefore, a slot creation whose status
>> check interleaved with the deactivation could continue based on a
>> stale status. This affects two paths:
>>
>> For regular slot creation, EnsureLogicalDecodingEnabled() assumed that
>> logical decoding must still be enabled during recovery since the
>> caller had already checked it. If a promotion interleaves as described
>>  above, the backend creating the slot fails with:
>>
>>     TRAP: failed Assert("IsLogicalDecodingEnabled()"), File: "logicalctl.c"
>>
>> For slot synchronization, the local slot could be created and
>> persisted based on the remote slot information fetched before the
>> deactivation was replayed, leaving a valid slot whose restart_lsn
>> precedes the deactivation. Decoding such a slot after a failover fails
>>  with:
>>
>>     ERROR:  unexpected logical decoding status change 0
>>
>> These races are confined to the narrow window between checking the
>> logical decoding status and the new slot becoming visible; once the
>> slot is visible, the invalidation performed by the deactivation
>> already covers it. So the fix is simple: re-check the logical decoding
>> status after the new slot becomes visible. Regular slot creation
>> raises an error and slot synchronization skips persisting the slot. If
>> the deactivation happens after the recheck instead, it is guaranteed
>> to invalidate the now-visible slot as usual. The attached 0002
>> implements this.
>
>
> i have looked into these conditions and they make sense and reviewed the
> v3-0002 patch, LGTM.

Thank you for reviewing the patch!

>
> while reviewing this, I had a thought (it's not related to these race issues), but
> if we disable logical decoding in primary by removing all the slots when
> wal_level = replica; it directly invalidates the private slots of the standby which
> seems unfair cause there might be some consumers using it, but then suddenly
> they get an error to either change the wal_level = logical on primary or add a slot
> on the primary, but instead i think we can make primary aware of the private
> slots of standby and keep logical decoding on, during the
> XLOG_LOGICAL_DECODING_STATUS_CHANGE record redo, thoughts?
>

IIUC it's too late to inform the primary during
XLOG_LOGICAL_DECODING_STATUS_CHANGE redo; the standby replays that
record only after the primary has disabled logical decoding, so WAL
lacking the information required for logical decoding has already been
generated. Once such a gap exists, the standby's slots cannot decode
past it even if the primary re-enabled logical decoding in response,
so we would have to invalidate them anyway.

Alternatively, standbys could proactively tell the primary about their
slots (like hot_standby_feedback), but I don't think this can be made
reliable. With cascaded standbys the information has to be propagated
up through each level, and the propagation lag leaves an unavoidable
race: by the time the primary learns that a downstream server still
needs logical WAL, its slots may already be gone, or a new slot could
be created right after the primary decided to disable.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Re: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

On Mon, Sep 21, 2026 at 11:32 PM Nikolay Samokhvalov  wrote:
>
> On Thu, Jul 16, 2026 at 6:52 AM Masahiko Sawada
>  wrote:
> > For slot synchronization, the local slot could be created and
> > persisted based on the remote slot information fetched before the
> > deactivation was replayed, leaving a valid slot whose restart_lsn
> > precedes the deactivation. Decoding such a slot after a failover fails
> > with:
> >
> > ERROR: unexpected logical decoding status change 0
> >
> > These races are confined to the narrow window between checking the
> > logical decoding status and the new slot becoming visible; once the
> > slot is visible, the invalidation performed by the deactivation
> > already covers it. So the fix is simple: re-check the logical decoding
> > status after the new slot becomes visible. Regular slot creation
> > raises an error and slot synchronization skips persisting the slot. If
> > the deactivation happens after the recheck instead, it is guaranteed
> > to invalidate the now-visible slot as usual. The attached 0002
> > implements this.
>
> The disable/re-enable case described in the comment above the final
> IsLogicalDecodingEnabled() check in update_and_persist_local_synced_slot() is
> reachable.
>
> On b73d13c3, the reproducer uses this sequence:
>
> 1. Slot sync fetches failover slot S and pauses at
> replication-slot-create-begin, before creating the local slot.
> 2. The primary drops S. The standby replays the logical-decoding
> deactivation while no local S exists to invalidate.
> 3. The primary recreates S. The standby replays the reactivation.
> 4. The old slot sync resumes with the first incarnation's restart_lsn.
>
> The final IsLogicalDecodingEnabled() check now returns true, so the old slot
> information is persisted. After promoting the standby, decoding that slot
> fails with:
>
> ERROR: unexpected logical decoding status change 0

Thank you for the report. Yes, while the window is very short in
practice, it indeed happens if the logical decoding is disabled and
re-enabled (by dropping and creating the same name failover slot)
between the slotsync worker fetches the slot information and creates
it.

It actually hits my concern mentioned in the comment in
update_and_persist_local_synced_slot():

     * XXX: this check cannot detect the case where logical decoding is
     * already re-enabled by a slot creation on the primary at this point.
     * Detecting that would require comparing the slot's restart_lsn with the
     * LSN at which logical decoding was last enabled.

> The attached patch adds a logical-decoding status generation. Slot sync
> records it before fetching remote slot information and refuses to persist a
> new slot if the generation changed in the meantime. It drops the temporary
> slot so that the next attempt fetches the current incarnation.

 Thank you for the patch.

An alternative approach that I think is better is to have the LSN of
the last replayed status change record in LogicalDecodingCtlData, and
check if logical decoding has been enabled since the remote slot's
restart_lsn. That's simpler than the proposed approach as we don't
need to increment the generation counter at both activation and
deactivation (which is not necessary outside recovery), nor to add
logical_decoding_generation to RemoteSlot. It also checks what we
actually need, that is, whether the WAL from the restart_lsn can be
decoded, rather than whether the status changed while synchronizing
slots.

Also, I think it's better to move the check to right after
ReplicationSlotCreate() in synchronize_one_slot() because (1) it can
simplify the code flow as we don't need to care about the slot dropped
in update_and_persist_local_synced_slot(), (2) it can save the WAL
reservation and the xmin_horizon computation, and (3) IIUC with the
proposed patch, the check can be bypassed when
update_and_persist_local_synced_slot() returns early due to
slotsync_skip_reason, leaving a temporary slot with the stale
restart_lsn. Once the slot passes the check right after its creation,
a later deactivation invalidates the slot, so we don't need to check
it again before persisting the slot.

I've attached the patch.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Re: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

Re: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

Re: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

Fwd: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

Re: Fix "unexpected logical decoding status change" error; from concurrent logical decoding activation

On Thu, Jul 16, 2026 at 6:52 AM Masahiko Sawada
 wrote:
> For slot synchronization, the local slot could be created and
> persisted based on the remote slot information fetched before the
> deactivation was replayed, leaving a valid slot whose restart_lsn
> precedes the deactivation. Decoding such a slot after a failover fails
> with:
>
> ERROR: unexpected logical decoding status change 0
>
> These races are confined to the narrow window between checking the
> logical decoding status and the new slot becoming visible; once the
> slot is visible, the invalidation performed by the deactivation
> already covers it. So the fix is simple: re-check the logical decoding
> status after the new slot becomes visible. Regular slot creation
> raises an error and slot synchronization skips persisting the slot. If
> the deactivation happens after the recheck instead, it is guaranteed
> to invalidate the now-visible slot as usual. The attached 0002
> implements this.

The disable/re-enable case described in the comment above the final
IsLogicalDecodingEnabled() check in update_and_persist_local_synced_slot() is
reachable.

On b73d13c3, the reproducer uses this sequence:

1. Slot sync fetches failover slot S and pauses at
replication-slot-create-begin, before creating the local slot.
2. The primary drops S. The standby replays the logical-decoding
deactivation while no local S exists to invalidate.
3. The primary recreates S. The standby replays the reactivation.
4. The old slot sync resumes with the first incarnation's restart_lsn.

The final IsLogicalDecodingEnabled() check now returns true, so the old slot
information is persisted. After promoting the standby, decoding that slot
fails with:

ERROR: unexpected logical decoding status change 0

The attached patch adds a logical-decoding status generation. Slot sync
records it before fetching remote slot information and refuses to persist a
new slot if the generation changed in the meantime. It drops the temporary
slot so that the next attempt fetches the current incarnation.

The new injection-point test fails without the fix because the stale slot is
persisted. With the fix, it verifies that the replacement slot is fetched
and that decoding succeeds after promotion. The existing
051_effective_wal_level test and the core regression tests also pass.

This work was done by our new AI harness for Postgres testing.

Nik