Re: [HACKERS] Restricting maximum keep segments by repslots

Hello,

I'll add this to CF2017-09.

At Mon, 06 Mar 2017 18:20:06 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote in
<20170306.182006.172683338.horiguchi.kyotaro@lab.ntt.co.jp>
> Thank you for the comment.
> 
> At Fri, 3 Mar 2017 14:47:20 -0500, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote in
<ac510b45-7805-7ccc-734c-1b38a6645f3e@2ndquadrant.com>
> > On 3/1/17 19:54, Kyotaro HORIGUCHI wrote:
> > >> Please measure it in size, not in number of segments.
> > > It was difficult to dicide which is reaaonable but I named it
> > > after wal_keep_segments because it has the similar effect.
> > > 
> > > In bytes(or LSN)
> > >  max_wal_size
> > >  min_wal_size
> > >  wal_write_flush_after
> > > 
> > > In segments
> > >  wal_keep_segments
> > 
> > We have been moving away from measuring in segments.  For example,
> > checkpoint_segments was replaced by max_wal_size.
> > 
> > Also, with the proposed patch that allows changing the segment size more
> > easily, this will become more important.  (I wonder if that will require
> > wal_keep_segments to change somehow.)
> 
> Agreed. It is 'max_slot_wal_keep_size' in the new version.
> 
> wal_keep_segments might should be removed someday.

- Following to min/max_wal_size, the variable was renamed to "max_slot_wal_keep_size_mb" and used as
ConvertToXSegs(x)"

- Stopped warning when checkpoint doesn't flush segments required by slots even if max_slot_wal_keep_size have worked.

- Avoided subraction that may be negative.

regards,

*** a/src/backend/access/transam/xlog.c
--- b/src/backend/access/transam/xlog.c
***************
*** 105,110 **** int            wal_level = WAL_LEVEL_MINIMAL;
--- 105,111 ---- int            CommitDelay = 0;    /* precommit delay in microseconds */ int            CommitSiblings
=5; /* # concurrent xacts needed to sleep */ int            wal_retrieve_retry_interval = 5000;
 
+ int            max_slot_wal_keep_size_mb = 0;  #ifdef WAL_DEBUG bool        XLOG_DEBUG = false;
***************
*** 9353,9361 **** KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
--- 9354,9385 ----     if (max_replication_slots > 0 && keep != InvalidXLogRecPtr)     {         XLogSegNo
slotSegNo;
+         int            slotlimitsegs = ConvertToXSegs(max_slot_wal_keep_size_mb);          XLByteToSeg(keep,
slotSegNo);
 
+         /*
+          * ignore slots if too many wal segments are kept.
+          * max_slot_wal_keep_size is just accumulated on wal_keep_segments.
+          */
+         if (max_slot_wal_keep_size_mb > 0 && slotSegNo + slotlimitsegs < segno)
+         {
+             segno = segno - slotlimitsegs; /* must be positive */
+ 
+             /*
+              * warn only if the checkpoint flushes the required segment.
+              * we assume here that *logSegNo is calculated keep location.
+              */
+             if (slotSegNo < *logSegNo)
+                 ereport(WARNING,
+                     (errmsg ("restart LSN of replication slots is ignored by checkpoint"),
+                      errdetail("Some replication slots have lost required WAL segnents to continue by up to %ld
segments.",
+                        (segno < *logSegNo ? segno : *logSegNo) - slotSegNo)));
+ 
+             /* emergency vent */
+             slotSegNo = segno;
+         }
+          if (slotSegNo <= 0)             segno = 1;         else if (slotSegNo < segno)
*** a/src/backend/utils/misc/guc.c
--- b/src/backend/utils/misc/guc.c
***************
*** 2366,2371 **** static struct config_int ConfigureNamesInt[] =
--- 2366,2382 ----     },      {
+         {"max_slot_wal_keep_size", PGC_SIGHUP, REPLICATION_SENDING,
+             gettext_noop("Sets the maximum size of extra WALs kept by replication slots."),
+          NULL,
+          GUC_UNIT_MB
+         },
+         &max_slot_wal_keep_size_mb,
+         0, 0, INT_MAX,
+         NULL, NULL, NULL
+     },
+ 
+     {         {"wal_sender_timeout", PGC_SIGHUP, REPLICATION_SENDING,             gettext_noop("Sets the maximum time
towait for WAL replication."),             NULL,
 
*** a/src/backend/utils/misc/postgresql.conf.sample
--- b/src/backend/utils/misc/postgresql.conf.sample
***************
*** 235,240 ****
--- 235,241 ---- #max_wal_senders = 10        # max number of walsender processes                 # (change requires
restart)#wal_keep_segments = 0        # in logfile segments, 16MB each; 0 disables
 
+ #max_slot_wal_keep_size = 0    # measured in bytes; 0 disables #wal_sender_timeout = 60s    # in milliseconds; 0
disables #max_replication_slots = 10    # max number of replication slots
 
*** a/src/include/access/xlog.h
--- b/src/include/access/xlog.h
***************
*** 97,102 **** extern bool reachedConsistency;
--- 97,103 ---- extern int    min_wal_size_mb; extern int    max_wal_size_mb; extern int    wal_keep_segments;
+ extern int    max_slot_wal_keep_size_mb; extern int    XLOGbuffers; extern int    XLogArchiveTimeout; extern int
wal_retrieve_retry_interval;