On 02/07/2018 10:22 PM, Colin Morelli wrote:
> Hey Tomas,
>
> We're using v1.1.0 of pglogical. I can't honestly say what could
> reproduce this issue as I'm not entirely sure and haven't tested it yet.
>
Well, consider updating - there's 1.1.2 in that exact branch, and
there's also 2.1.1 now. I don't recall if there were bugs with such
symptoms, though.
> I had thought it was caused when a toasted row is deleted before the
> pglogical_output plugin processes the event (so it fails when going to
> de-toast the row), but I would imagine that postgres internally should
> prevent deleting data that would be needed by a replication stream. I
> must admit I'm not well versed in this area, though. My understanding is
> wal2json doesn't detoast the data at all, and thus doesn't have an issue
> (for our case, we really just need the json data with an identifier
> telling us that the row changed, we're not doing full logical replication).
>
That seems really unlikely, for a couple of reasons.
1) As you've correctly guessed, toasted values are protected by the same
MVCC mechanism as regular tables, so rows should not disappear when
transactions might still need them (including replication).
2) The detoasting happens in reorderbuffer, which is shared both by
pglogical and wal2json. So both should have been affected.
3) You've also mentioned missing files in base/ which suggests there's
something far more serious going on. Not sure what, though.
I suggest you do one or both of these things:
a) update to newer pglogical version and see if it fixes the issue
b) try preparing a self-contained reproducer
c) share a more complete version of the server logs (hard to say what's
the issue from just a few lines)
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services