Re: prevent immature WAL streaming

At Wed, 25 Aug 2021 18:18:59 +0000, "Bossart, Nathan" <bossartn@amazon.com> wrote in 
> On 8/25/21, 5:33 AM, "alvherre@alvh.no-ip.org" <alvherre@alvh.no-ip.org> wrote:
> > On 2021-Aug-24, Bossart, Nathan wrote:
> >> Another interesting thing I see is that the boundary stored in
> >> earliestSegBoundary is not necessarily the earliest one.  It's just
> >> the first one that has been registered.  I did this for simplicity for
> >> the .ready file fix, but I can see it causing problems here.
> >
> > Hmm, is there really a problem here?  Surely the flush point cannot go
> > past whatever has been written.  If somebody is writing an earlier
> > section of WAL, then we cannot move the flush pointer to a later
> > position.  So it doesn't matter if the earliest point we have registered
> > is the true earliest -- we only care for it to be the earliest that is
> > past the flush point.
> 
> Let's say we have the following situation (F = flush, E = earliest
> registered boundary, and L = latest registered boundary), and let's
> assume that each segment has a cross-segment record that ends in the
> next segment.
> 
>         F     E                                         L
>         |-----|-----|-----|-----|-----|-----|-----|-----|
>            1     2     3     4     5     6     7     8
> 
> Then, we write out WAL to disk and create .ready files as needed.  If
> we didn't flush beyond the latest registered boundary, the latest
> registered boundary now becomes the earliest boundary.
> 
>                           F                             E
>         |-----|-----|-----|-----|-----|-----|-----|-----|
>            1     2     3     4     5     6     7     8
> 
> At this point, the earliest segment boundary past the flush point is
> before the "earliest" boundary we are tracking.

We know we have some cross-segment records in the regin [E L] so we
cannot add a .ready file if flush is in the region. I haven't looked
the latest patch (or I may misunderstand the discussion here) but I
think we shouldn't move E before F exceeds previous (or in the first
picture above) L.  Things are done that way in my ancient proposal in
[1].


https://www.postgresql.org/message-id/attachment/117052/v4-0001-Avoid-archiving-a-WAL-segment-that-continues-to-t.patch
+    if (LogwrtResult.Write < firstSegContRecStart ||
+        lastSegContRecEnd <= LogwrtResult.Write)
+    {
        <notify the last segment>

By doing so, at the time of the second picutre, the pointers are set as:

               E           F                             L
         |-----|-----|-----|-----|-----|-----|-----|-----|
            1     2     3     4     5     6     7     8

Then the poiter are cleard at the time F reaches L,

                                                         F
         |-----|-----|-----|-----|-----|-----|-----|-----|
            1     2     3     4     5     6     7     8

Isn't this work correctly?  As I think I mentioned in the thread, I
don't think we don't have so many (more than several, specifically)
segments in a region [E L].

[1] https://www.postgresql.org/message-id/20201216.110120.887433782054853494.horikyota.ntt%40gmail.com

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center