Hi Euler!
Thanks for your response.
> 4 дек. 2020 г., в 22:14, Euler Taveira <euler.taveira@2ndquadrant.com> написал(а):
>
> On Fri, 4 Dec 2020 at 04:33, Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>
> I was discussing problems of CDC with scientific community and they asked this simple question: "So you have
efficientWAL archive on a very cheap storage, why don't you have a logical archive too?"
>
> WAL archive doesn't process data; it just copies from one location into another one. However, "logical archive" must
processdata.
WAL archiving processes data: it does compression, encryption and digesting. Only minimal impractical setup will copy
dataas is. However I agree, that all processing is done outside postgres.
> If we could just run archive command ```archive-tool wal-push 0000000900000F2C000000E1.logical``` with contents of
logicalreplication - this would be super cool for OLAP. I'd prefer even avoid writing 0000000900000F2C000000E1.logical
todisk, i.e. push data on stdio or something like that.
>
> The most time consuming process is logical decoding, mainly due to long running transactions.
Currently I do not experience problem of high CPU utilisation.
> In order to minimize your issue, we should improve the logical decoding mechanism.
No, the issue I'm facing comes from the fact that corner cases of failover are not solved properly for logical
replication.Timelines, partial segments, archiving along with streaming, starting from arbitrary LSN (within available
WAL),rewind, named restore points, cascade replication etc etc. All these nice things are there for WAL and are missing
forLR. I'm just trying to find shortest path through this to make CDC(changed data capture) work.
> There was a discussion about allowing logical decoding on the replica that would probably help your use case a lot.
I will look there more closely, thanks! But it's only part of a solution.
Best regards, Andrey Borodin.