Re: Protect syscache from bloating with negative cache entries

This is a new complete workable patch after a long time of struggling
with benchmarking.

At Tue, 19 Nov 2019 19:48:10 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in 
> I ran the run2.sh script attached, which runs catcachebench2(), which
> asks SearchSysCache3() for cached entries (almost) 240000 times per
> run.  The number of each output line is the mean of 3 times runs, and
> stddev. Lines are in "time" order and edited to fit here. "gen_tbl.pl
> | psql" creates a database for the benchmark. catcachebench2() runs
> the shortest path in the three in the attached benchmark program.
> 
> (pg_ctl start)
> $ perl gen_tbl.pl | psql ...
> (pg_ctl stop)

I wonder why I took the average of the time instead of choose the
fastest one.  This benchmark is extremely CPU intensive so the fastest
run reliably represents the perfromance.

I changed the benchmark so that it shows the time of the fastest run
(run4.sh). Based on the latest result, I used the pattern 3
(SearchSyscacheN indirection, but wrongly pointed as 1 in the last
mail) in the latest version,

I took the number of the fastest time among 3 iteration of 5 runs of
both master/patched O2 binaries.

 version |   min   
---------+---------
 master  | 7986.65 
 patched | 7984.47   = 'indirect' below

I would say this version doesn't get degradaded by indirect calls.
So, I applied the other part of the catcache expiration patch as the
succeeding parts. After that I got somewhat strange but very stable
result.  Just adding struct members acceleartes the benchmark. The
numbers are the fastest time of 20 runs of the bencmark in 10 times
iterations.

              ms
master      7980.79   # the master with the benchmark extension (0001)
=====
base        7340.96   # add only struct members and a GUC variable. (0002)
indirect    7998.68   # call SearchCatCacheN indirectly (0003)
=====
expire-off  7422.30   # CatCache expiration (0004)
                      # (catalog_cache_prune_min_age = -1)
expire-on   7861.13   # CatCache expiration (catalog_cache_prune_min_age = 0)


The patch accelerates CatCaCheSearch for uncertain reasons. I'm not
sure what makes the difference between about 8000ms and about 7400 ms,
though. Several times building of all versions then running the
benchmark gave me the results with the same tendency. I once stop this
work at this point and continue later. The following files are
attached.

0001-catcache-benchmark-extension.patch:
  benchmnark extension used by the benchmarking here.  The test tables
  are generated using gentbl2.pl attached. (perl gentbk2.pl | psql)

0002-base_change.patch:
  Preliminary adds some struct members and a GUC variable to see if
  they cause any extent of degradation.

0003-Make-CatCacheSearchN-indirect-functions.patch:
  Rewrite to change CatCacheSearchN functions to be called indirectly.

0004-CatCache-expiration-feature.patch:
  Add CatCache expiration feature.

gentbl2.pl: A script that emits SQL statements to generate test tables.
run4.sh   : The test script I used for benchmarkiing here.
build2.sh : A script I used to build the four types of binaries used here.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center
From dacf4a2ac9eb49099e744ee24066b94e9f78aa61 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Thu, 14 Nov 2019 19:24:36 +0900
Subject: [PATCH 1/4] catcache benchmark extension

Provides the function catcachebench(bench_no int), which runs CPU
intensive benchmark on catcache search. The test table is created by a
script separately provided.

catcachebench(0): prewarm catcache with provided test tables.
catcachebench(1): fetches all attribute stats of all tables.
    This benchmark loads a vast number of unique entries.
    Expriration doesn't work since it runs in a transaction.
catcachebench(2): fetches all attribute stats of a tables many times.
    This benchmark repeatedly accesses already loaded entries.
    Expriration doesn't work since it runs in a transaction.
catcachebench(3): fetches all attribute stats of all tables four times.
    Different from other modes, this runs expiration by forcibly
    updates reference clock variable every 1000 entries.

At this point, variables needed for the expiration feature is not
added so SetCatCacheClock is a dummy macro that just replaces it with
its parameter.
---
 contrib/catcachebench/Makefile               |  17 +
 contrib/catcachebench/catcachebench--0.0.sql |  14 +
 contrib/catcachebench/catcachebench.c        | 330 +++++++++++++++++++
 contrib/catcachebench/catcachebench.control  |   6 +
 src/backend/utils/cache/catcache.c           |  35 ++
 src/backend/utils/cache/syscache.c           |   2 +-
 src/include/utils/catcache.h                 |   3 +
 7 files changed, 406 insertions(+), 1 deletion(-)
 create mode 100644 contrib/catcachebench/Makefile
 create mode 100644 contrib/catcachebench/catcachebench--0.0.sql
 create mode 100644 contrib/catcachebench/catcachebench.c
 create mode 100644 contrib/catcachebench/catcachebench.control

diff --git a/contrib/catcachebench/Makefile b/contrib/catcachebench/Makefile
new file mode 100644
index 0000000000..0478818b25
--- /dev/null
+++ b/contrib/catcachebench/Makefile
@@ -0,0 +1,17 @@
+MODULE_big = catcachebench
+OBJS = catcachebench.o
+
+EXTENSION = catcachebench
+DATA = catcachebench--0.0.sql
+PGFILEDESC = "catcachebench - benchmark for catcache pruning feature"
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/catcachebench
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/catcachebench/catcachebench--0.0.sql b/contrib/catcachebench/catcachebench--0.0.sql
new file mode 100644
index 0000000000..ea9cd62abb
--- /dev/null
+++ b/contrib/catcachebench/catcachebench--0.0.sql
@@ -0,0 +1,14 @@
+/* contrib/catcachebench/catcachebench--0.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION catcachebench" to load this file. \quit
+
+CREATE FUNCTION catcachebench(IN type int)
+RETURNS double precision
+AS 'MODULE_PATHNAME', 'catcachebench'
+LANGUAGE C STRICT VOLATILE;
+
+CREATE FUNCTION catcachereadstats(OUT catid int, OUT reloid oid, OUT searches bigint, OUT hits bigint, OUT neg_hits
bigint)
+RETURNS SETOF record
+AS 'MODULE_PATHNAME', 'catcachereadstats'
+LANGUAGE C STRICT VOLATILE;
diff --git a/contrib/catcachebench/catcachebench.c b/contrib/catcachebench/catcachebench.c
new file mode 100644
index 0000000000..b6c2b8f577
--- /dev/null
+++ b/contrib/catcachebench/catcachebench.c
@@ -0,0 +1,330 @@
+/*
+ * catcachebench: test code for cache pruning feature
+ */
+/* #define CATCACHE_STATS */
+#include "postgres.h"
+#include "catalog/pg_type.h"
+#include "catalog/pg_statistic.h"
+#include "executor/spi.h"
+#include "funcapi.h"
+#include "libpq/pqsignal.h"
+#include "utils/catcache.h"
+#include "utils/syscache.h"
+#include "utils/timestamp.h"
+
+Oid        tableoids[10000];
+int        ntables = 0;
+int16    attnums[1000];
+int        natts = 0;
+
+PG_MODULE_MAGIC;
+
+double catcachebench1(void);
+double catcachebench2(void);
+double catcachebench3(void);
+void collectinfo(void);
+void catcachewarmup(void);
+
+PG_FUNCTION_INFO_V1(catcachebench);
+PG_FUNCTION_INFO_V1(catcachereadstats);
+
+extern void CatalogCacheFlushCatalog2(Oid catId);
+extern int64 catcache_called;
+extern CatCache *SysCache[];
+
+typedef struct catcachestatsstate
+{
+    TupleDesc tupd;
+    int          catId;
+} catcachestatsstate;
+
+Datum
+catcachereadstats(PG_FUNCTION_ARGS)
+{
+    catcachestatsstate *state_data = NULL;
+    FuncCallContext *fctx;
+
+    if (SRF_IS_FIRSTCALL())
+    {
+        TupleDesc    tupdesc;
+        MemoryContext mctx;
+
+        fctx = SRF_FIRSTCALL_INIT();
+        mctx = MemoryContextSwitchTo(fctx->multi_call_memory_ctx);
+
+        state_data = palloc(sizeof(catcachestatsstate));
+
+        /* Build a tuple descriptor for our result type */
+        if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+            elog(ERROR, "return type must be a row type");
+
+        state_data->tupd = tupdesc;
+        state_data->catId = 0;
+
+        fctx->user_fctx = state_data;
+
+        MemoryContextSwitchTo(mctx);
+    }
+
+    fctx = SRF_PERCALL_SETUP();
+    state_data = fctx->user_fctx;
+
+    if (state_data->catId < SysCacheSize)
+    {
+        Datum    values[5];
+        bool    nulls[5];
+        HeapTuple    resulttup;
+        Datum    result;
+        int        catId = state_data->catId++;
+
+        memset(nulls, 0, sizeof(nulls));
+        memset(values, 0, sizeof(values));
+        values[0] = Int16GetDatum(catId);
+        values[1] = ObjectIdGetDatum(SysCache[catId]->cc_reloid);
+#ifdef CATCACHE_STATS        
+        values[2] = Int64GetDatum(SysCache[catId]->cc_searches);
+        values[3] = Int64GetDatum(SysCache[catId]->cc_hits);
+        values[4] = Int64GetDatum(SysCache[catId]->cc_neg_hits);
+#endif
+        resulttup = heap_form_tuple(state_data->tupd, values, nulls);
+        result = HeapTupleGetDatum(resulttup);
+
+        SRF_RETURN_NEXT(fctx, result);
+    }
+
+    SRF_RETURN_DONE(fctx);
+}
+
+Datum
+catcachebench(PG_FUNCTION_ARGS)
+{
+    int        testtype = PG_GETARG_INT32(0);
+    double    ms;
+
+    collectinfo();
+
+    /* flush the catalog -- safe? don't mind. */
+    CatalogCacheFlushCatalog2(StatisticRelationId);
+
+    switch (testtype)
+    {
+    case 0:
+        catcachewarmup(); /* prewarm of syscatalog */
+        PG_RETURN_NULL();
+    case 1:
+        ms = catcachebench1(); break;
+    case 2:
+        ms = catcachebench2(); break;
+    case 3:
+        ms = catcachebench3(); break;
+    default:
+        elog(ERROR, "Invalid test type: %d", testtype);
+    }
+
+    PG_RETURN_DATUM(Float8GetDatum(ms));
+}
+
+/*
+ * fetch all attribute entires of all tables.
+ */
+double
+catcachebench1(void)
+{
+    int t, a;
+    instr_time    start,
+                duration;
+
+    PG_SETMASK(&BlockSig);
+    INSTR_TIME_SET_CURRENT(start);
+    for (t = 0 ; t < ntables ; t++)
+    {
+        for (a = 0 ; a < natts ; a++)
+        {
+            HeapTuple tup;
+
+            tup = SearchSysCache3(STATRELATTINH,
+                                  ObjectIdGetDatum(tableoids[t]),
+                                  Int16GetDatum(attnums[a]),
+                                  BoolGetDatum(false));
+            /* should be null, but.. */
+            if (HeapTupleIsValid(tup))
+                ReleaseSysCache(tup);
+        }
+    }
+    INSTR_TIME_SET_CURRENT(duration);
+    INSTR_TIME_SUBTRACT(duration, start);
+    PG_SETMASK(&UnBlockSig);
+
+    return INSTR_TIME_GET_MILLISEC(duration);
+};
+
+/*
+ * fetch all attribute entires of a table many times.
+ */
+double
+catcachebench2(void)
+{
+    int t, a;
+    instr_time    start,
+                duration;
+
+    PG_SETMASK(&BlockSig);
+    INSTR_TIME_SET_CURRENT(start);
+    for (t = 0 ; t < 240000 ; t++)
+    {
+        for (a = 0 ; a < natts ; a++)
+        {
+            HeapTuple tup;
+
+            tup = SearchSysCache3(STATRELATTINH,
+                                  ObjectIdGetDatum(tableoids[0]),
+                                  Int16GetDatum(attnums[a]),
+                                  BoolGetDatum(false));
+            /* should be null, but.. */
+            if (HeapTupleIsValid(tup))
+                ReleaseSysCache(tup);
+        }
+    }
+    INSTR_TIME_SET_CURRENT(duration);
+    INSTR_TIME_SUBTRACT(duration, start);
+    PG_SETMASK(&UnBlockSig);
+
+    return INSTR_TIME_GET_MILLISEC(duration);
+};
+
+/*
+ * fetch all attribute entires of all tables several times with having
+ * expiration happen.
+ */
+double
+catcachebench3(void)
+{
+    const int clock_step = 1000;
+    int i, t, a;
+    instr_time    start,
+                duration;
+
+    PG_SETMASK(&BlockSig);
+    INSTR_TIME_SET_CURRENT(start);
+    for (i = 0 ; i < 4 ; i++)
+    {
+        int ct = clock_step;
+
+        for (t = 0 ; t < ntables ; t++)
+        {
+            /*
+             * catcacheclock is updated by transaction timestamp, so needs to
+             * be updated by other means for this test to work. Here I choosed
+             * to update the clock every 1000 tables scan.
+             */
+            if (--ct < 0)
+            {
+                SetCatCacheClock(GetCurrentTimestamp());
+                ct = clock_step;
+            }
+            for (a = 0 ; a < natts ; a++)
+            {
+                HeapTuple tup;
+
+                tup = SearchSysCache3(STATRELATTINH,
+                                      ObjectIdGetDatum(tableoids[t]),
+                                      Int16GetDatum(attnums[a]),
+                                      BoolGetDatum(false));
+                /* should be null, but.. */
+                if (HeapTupleIsValid(tup))
+                    ReleaseSysCache(tup);
+            }
+        }
+    }
+    INSTR_TIME_SET_CURRENT(duration);
+    INSTR_TIME_SUBTRACT(duration, start);
+    PG_SETMASK(&UnBlockSig);
+
+    return INSTR_TIME_GET_MILLISEC(duration);
+};
+
+void
+catcachewarmup(void)
+{
+    int t, a;
+
+    /* load up catalog tables */
+    for (t = 0 ; t < ntables ; t++)
+    {
+        for (a = 0 ; a < natts ; a++)
+        {
+            HeapTuple tup;
+
+            tup = SearchSysCache3(STATRELATTINH,
+                                  ObjectIdGetDatum(tableoids[t]),
+                                  Int16GetDatum(attnums[a]),
+                                  BoolGetDatum(false));
+            /* should be null, but.. */
+            if (HeapTupleIsValid(tup))
+                ReleaseSysCache(tup);
+        }
+    }
+}
+
+void
+collectinfo(void)
+{
+    int ret;
+    Datum    values[10000];
+    bool    nulls[10000];
+    Oid        types0[] = {OIDOID};
+    int i;
+
+    ntables = 0;
+    natts = 0;
+
+    SPI_connect();
+    /* collect target tables */
+    ret = SPI_execute("select oid from pg_class where relnamespace = (select oid from pg_namespace where nspname =
\'test\')",
+                      true, 0);
+    if (ret != SPI_OK_SELECT)
+        elog(ERROR, "Failed 1");
+    if (SPI_processed == 0)
+        elog(ERROR, "no relation found in schema \"test\"");
+    if (SPI_processed > 10000)
+        elog(ERROR, "too many relation found in schema \"test\"");
+
+    for (i = 0 ; i < SPI_processed ; i++)
+    {
+        heap_deform_tuple(SPI_tuptable->vals[i], SPI_tuptable->tupdesc,
+                          values, nulls);
+        if (nulls[0])
+            elog(ERROR, "Failed 2");
+
+        tableoids[ntables++] = DatumGetObjectId(values[0]);
+    }
+    SPI_finish();
+    elog(DEBUG1, "%d tables found", ntables);
+
+    values[0] = ObjectIdGetDatum(tableoids[0]);
+    nulls[0] = false;
+    SPI_connect();
+    ret = SPI_execute_with_args("select attnum from pg_attribute where attrelid = (select oid from pg_class where oid
=$1)",
 
+                                1, types0, values, NULL, true, 0);
+    if (SPI_processed == 0)
+        elog(ERROR, "no attribute found in table %d", tableoids[0]);
+    if (SPI_processed > 10000)
+        elog(ERROR, "too many relation found in table %d", tableoids[0]);
+    
+    /* collect target attributes. assuming all tables have the same attnums */
+    for (i = 0 ; i < SPI_processed ; i++)
+    {
+        int16 attnum;
+
+        heap_deform_tuple(SPI_tuptable->vals[i], SPI_tuptable->tupdesc,
+                          values, nulls);
+        if (nulls[0])
+            elog(ERROR, "Failed 3");
+        attnum = DatumGetInt16(values[0]);
+
+        if (attnum > 0)
+            attnums[natts++] = attnum;
+    }
+    SPI_finish();
+    elog(DEBUG1, "%d attributes found", natts);
+}
diff --git a/contrib/catcachebench/catcachebench.control b/contrib/catcachebench/catcachebench.control
new file mode 100644
index 0000000000..3fc9d2e420
--- /dev/null
+++ b/contrib/catcachebench/catcachebench.control
@@ -0,0 +1,6 @@
+# catcachebench
+
+comment = 'benchmark for catcache pruning'
+default_version = '0.0'
+module_pathname = '$libdir/catcachebench'
+relocatable = true
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 64776e3209..95a4e30d2b 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -740,6 +740,41 @@ CatalogCacheFlushCatalog(Oid catId)
     CACHE_elog(DEBUG2, "end of CatalogCacheFlushCatalog call");
 }
 
+
+/* FUNCTION FOR BENCHMARKING */
+void
+CatalogCacheFlushCatalog2(Oid catId)
+{
+    slist_iter    iter;
+
+    CACHE_elog(DEBUG2, "CatalogCacheFlushCatalog called for %u", catId);
+
+    slist_foreach(iter, &CacheHdr->ch_caches)
+    {
+        CatCache   *cache = slist_container(CatCache, cc_next, iter.cur);
+
+        /* Does this cache store tuples of the target catalog? */
+        if (cache->cc_reloid == catId)
+        {
+            /* Yes, so flush all its contents */
+            ResetCatalogCache(cache);
+
+            /* Tell inval.c to call syscache callbacks for this cache */
+            CallSyscacheCallbacks(cache->id, 0);
+
+            cache->cc_nbuckets = 128;
+            pfree(cache->cc_bucket);
+            cache->cc_bucket = palloc0(128 * sizeof(dlist_head));
+            ereport(DEBUG1,
+                    (errmsg("Catcache reset"),
+                     errhidestmt(true)));
+        }
+    }
+
+    CACHE_elog(DEBUG2, "end of CatalogCacheFlushCatalog call");
+}
+/* END: FUNCTION FOR BENCHMARKING */
+
 /*
  *        InitCatCache
  *
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index 53d9ddf159..1c79a85a8c 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -983,7 +983,7 @@ static const struct cachedesc cacheinfo[] = {
     }
 };
 
-static CatCache *SysCache[SysCacheSize];
+CatCache *SysCache[SysCacheSize];
 
 static bool CacheInitialized = false;
 
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index f4aa316604..ea9e75a1ae 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -228,4 +228,7 @@ extern void PrepareToInvalidateCacheTuple(Relation relation,
 extern void PrintCatCacheLeakWarning(HeapTuple tuple);
 extern void PrintCatCacheListLeakWarning(CatCList *list);
 
+/* tentative change to allow benchmark on master branch */
+#define SetCatCacheClock(ts) (ts)
+
 #endif                            /* CATCACHE_H */
-- 
2.23.0

From a18c8f531c685682b22d304efa8bfb31401cc3b0 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Fri, 10 Jan 2020 15:02:26 +0900
Subject: [PATCH 2/4] base_change

Adds struct members needed by catcache expiration feature and a GUC
variable that controls the behavior of the feature. But no substantial
code is not added yet. This also replaces SetCatCacheClock() with the
real definition.

If existence of some variables alone can cause degradation,
benchmarking after this patch shows that.
---
 src/backend/utils/cache/catcache.c | 15 +++++++++++++++
 src/backend/utils/misc/guc.c       | 13 +++++++++++++
 src/include/utils/catcache.h       | 23 ++++++++++++++++++++---
 3 files changed, 48 insertions(+), 3 deletions(-)

diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 95a4e30d2b..d267e5ce6e 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -60,9 +60,18 @@
 #define CACHE_elog(...)
 #endif
 
+/*
+ * GUC variable to define the minimum age of entries that will be considered
+ * to be evicted in seconds. -1 to disable the feature.
+ */
+int catalog_cache_prune_min_age = 300;
+
 /* Cache management header --- pointer is NULL until created */
 static CatCacheHeader *CacheHdr = NULL;
 
+/* Clock for the last accessed time of a catcache entry. */
+TimestampTz    catcacheclock = 0;
+
 static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
                                                int nkeys,
                                                Datum v1, Datum v2,
@@ -99,6 +108,12 @@ static void CatCacheFreeKeys(TupleDesc tupdesc, int nkeys, int *attnos,
 static void CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
                              Datum *srckeys, Datum *dstkeys);
 
+/* GUC assign function */
+void
+assign_catalog_cache_prune_min_age(int newval, void *extra)
+{
+    catalog_cache_prune_min_age = newval;
+}
 
 /*
  *                    internal support functions
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 62285792ec..2f2b599f61 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -83,6 +83,8 @@
 #include "tsearch/ts_cache.h"
 #include "utils/builtins.h"
 #include "utils/bytea.h"
+#include "utils/catcache.h"
+#include "utils/guc_tables.h"
 #include "utils/float.h"
 #include "utils/guc_tables.h"
 #include "utils/memutils.h"
@@ -2280,6 +2282,17 @@ static struct config_int ConfigureNamesInt[] =
         NULL, NULL, NULL
     },
 
+    {
+        {"catalog_cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+            gettext_noop("System catalog cache entries that live unused for longer than this seconds are considered
forremoval."),
 
+            gettext_noop("The value of -1 turns off pruning."),
+            GUC_UNIT_S
+        },
+        &catalog_cache_prune_min_age,
+        300, -1, INT_MAX,
+        NULL, assign_catalog_cache_prune_min_age, NULL
+    },
+
     /*
      * We use the hopefully-safely-small value of 100kB as the compiled-in
      * default for max_stack_depth.  InitializeGUCOptions will increase it if
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index ea9e75a1ae..3d3870f05a 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
 
 #include "access/htup.h"
 #include "access/skey.h"
+#include "datatype/timestamp.h"
 #include "lib/ilist.h"
 #include "utils/relcache.h"
 
@@ -61,6 +62,7 @@ typedef struct catcache
     slist_node    cc_next;        /* list link */
     ScanKeyData cc_skey[CATCACHE_MAXKEYS];    /* precomputed key info for heap
                                              * scans */
+    TimestampTz    cc_oldest_ts;    /* timestamp of the oldest tuple in the hash */
 
     /*
      * Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,6 +121,8 @@ typedef struct catctup
     bool        dead;            /* dead but not yet removed? */
     bool        negative;        /* negative cache entry? */
     HeapTupleData tuple;        /* tuple management header */
+    unsigned int naccess;        /* # of access to this entry */
+    TimestampTz    lastaccess;        /* timestamp of the last usage */
 
     /*
      * The tuple may also be a member of at most one CatCList.  (If a single
@@ -189,6 +193,22 @@ typedef struct catcacheheader
 /* this extern duplicates utils/memutils.h... */
 extern PGDLLIMPORT MemoryContext CacheMemoryContext;
 
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int catalog_cache_prune_min_age;
+
+/* source clock for access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/* SetCatCacheClock - set catcache timestamp source clodk */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+    catcacheclock = ts;
+}
+
+
+extern void assign_catalog_cache_prune_min_age(int newval, void *extra);
+
 extern void CreateCacheMemoryContext(void);
 
 extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
@@ -228,7 +248,4 @@ extern void PrepareToInvalidateCacheTuple(Relation relation,
 extern void PrintCatCacheLeakWarning(HeapTuple tuple);
 extern void PrintCatCacheListLeakWarning(CatCList *list);
 
-/* tentative change to allow benchmark on master branch */
-#define SetCatCacheClock(ts) (ts)
-
 #endif                            /* CATCACHE_H */
-- 
2.23.0

From 5327bfd024ba9e5313cca39db4a1e986c299ca16 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Thu, 9 Jan 2020 19:22:18 +0900
Subject: [PATCH 3/4] Make CatCacheSearchN indirect functions

After some expriments showed that the best way to add a new feature to
the current CatCacheSearch path is making SearchCatCacheN functions
replacable using indirect calling. This patch does that.

If the change of how to call the functions alone causes degradataion,
benchmarking after this patch applied shows that.
---
 src/backend/utils/cache/catcache.c | 42 +++++++++++++++++++++++-------
 src/include/utils/catcache.h       | 40 ++++++++++++++++++++++++----
 2 files changed, 67 insertions(+), 15 deletions(-)

diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index d267e5ce6e..74c893ba4e 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -84,6 +84,15 @@ static pg_noinline HeapTuple SearchCatCacheMiss(CatCache *cache,
                                                 Datum v1, Datum v2,
                                                 Datum v3, Datum v4);
 
+static HeapTuple SearchCatCacheb(CatCache *cache,
+                                 Datum v1, Datum v2, Datum v3, Datum v4);
+static HeapTuple SearchCatCache1b(CatCache *cache, Datum v1);
+static HeapTuple SearchCatCache2b(CatCache *cache, Datum v1, Datum v2);
+static HeapTuple SearchCatCache3b(CatCache *cache,
+                                  Datum v1, Datum v2, Datum v3);
+static HeapTuple SearchCatCache4b(CatCache *cache,
+                                  Datum v1, Datum v2, Datum v3, Datum v4);
+
 static uint32 CatalogCacheComputeHashValue(CatCache *cache, int nkeys,
                                            Datum v1, Datum v2, Datum v3, Datum v4);
 static uint32 CatalogCacheComputeTupleHashValue(CatCache *cache, int nkeys,
@@ -108,6 +117,16 @@ static void CatCacheFreeKeys(TupleDesc tupdesc, int nkeys, int *attnos,
 static void CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
                              Datum *srckeys, Datum *dstkeys);
 
+static SearchCatCacheFuncsType catcache_base = {
+    SearchCatCacheb,
+    SearchCatCache1b,
+    SearchCatCache2b,
+    SearchCatCache3b,
+    SearchCatCache4b
+};
+
+SearchCatCacheFuncsType *SearchCatCacheFuncs = NULL;
+
 /* GUC assign function */
 void
 assign_catalog_cache_prune_min_age(int newval, void *extra)
@@ -852,6 +871,9 @@ InitCatCache(int id,
         CacheHdr = (CatCacheHeader *) palloc(sizeof(CatCacheHeader));
         slist_init(&CacheHdr->ch_caches);
         CacheHdr->ch_ntup = 0;
+
+        SearchCatCacheFuncs = &catcache_base;
+
 #ifdef CATCACHE_STATS
         /* set up to dump stats at backend exit */
         on_proc_exit(CatCachePrintStats, 0);
@@ -1193,8 +1215,8 @@ IndexScanOK(CatCache *cache, ScanKey cur_skey)
  * the caller need not go to the trouble of converting it to a fully
  * null-padded NAME.
  */
-HeapTuple
-SearchCatCache(CatCache *cache,
+static HeapTuple
+SearchCatCacheb(CatCache *cache,
                Datum v1,
                Datum v2,
                Datum v3,
@@ -1210,32 +1232,32 @@ SearchCatCache(CatCache *cache,
  * bit faster than SearchCatCache().
  */
 
-HeapTuple
-SearchCatCache1(CatCache *cache,
+static HeapTuple
+SearchCatCache1b(CatCache *cache,
                 Datum v1)
 {
     return SearchCatCacheInternal(cache, 1, v1, 0, 0, 0);
 }
 
 
-HeapTuple
-SearchCatCache2(CatCache *cache,
+static HeapTuple
+SearchCatCache2b(CatCache *cache,
                 Datum v1, Datum v2)
 {
     return SearchCatCacheInternal(cache, 2, v1, v2, 0, 0);
 }
 
 
-HeapTuple
-SearchCatCache3(CatCache *cache,
+static HeapTuple
+SearchCatCache3b(CatCache *cache,
                 Datum v1, Datum v2, Datum v3)
 {
     return SearchCatCacheInternal(cache, 3, v1, v2, v3, 0);
 }
 
 
-HeapTuple
-SearchCatCache4(CatCache *cache,
+static HeapTuple
+SearchCatCache4b(CatCache *cache,
                 Datum v1, Datum v2, Datum v3, Datum v4)
 {
     return SearchCatCacheInternal(cache, 4, v1, v2, v3, v4);
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 3d3870f05a..f9e9889339 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -189,6 +189,36 @@ typedef struct catcacheheader
     int            ch_ntup;        /* # of tuples in all caches */
 } CatCacheHeader;
 
+typedef HeapTuple (*SearchCatCache_fn)(CatCache *cache,
+                                       Datum v1, Datum v2, Datum v3, Datum v4);
+typedef HeapTuple (*SearchCatCache1_fn)(CatCache *cache, Datum v1);
+typedef HeapTuple (*SearchCatCache2_fn)(CatCache *cache, Datum v1, Datum v2);
+typedef HeapTuple (*SearchCatCache3_fn)(CatCache *cache, Datum v1, Datum v2,
+                                        Datum v3);
+typedef HeapTuple (*SearchCatCache4_fn)(CatCache *cache,
+                                        Datum v1, Datum v2, Datum v3, Datum v4);
+
+typedef struct SearchCatCacheFuncsType
+{
+    SearchCatCache_fn    SearchCatCache;
+    SearchCatCache1_fn    SearchCatCache1;
+    SearchCatCache2_fn    SearchCatCache2;
+    SearchCatCache3_fn    SearchCatCache3;
+    SearchCatCache4_fn    SearchCatCache4;
+} SearchCatCacheFuncsType;
+
+extern PGDLLIMPORT SearchCatCacheFuncsType *SearchCatCacheFuncs;
+
+#define SearchCatCache(cache, v1, v2, v3, v4) \
+    SearchCatCacheFuncs->SearchCatCache(cache, v1, v2, v3, v4)
+#define SearchCatCache1(cache, v1) \
+    SearchCatCacheFuncs->SearchCatCache1(cache, v1)
+#define SearchCatCache2(cache, v1, v2) \
+    SearchCatCacheFuncs->SearchCatCache2(cache, v1, v2)
+#define SearchCatCache3(cache, v1, v2, v3) \
+    SearchCatCacheFuncs->SearchCatCache3(cache, v1, v2, v3)
+#define SearchCatCache4(cache, v1, v2, v3, v4) \
+    SearchCatCacheFuncs->SearchCatCache4(cache, v1, v2, v3, v4)
 
 /* this extern duplicates utils/memutils.h... */
 extern PGDLLIMPORT MemoryContext CacheMemoryContext;
@@ -216,15 +246,15 @@ extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
                               int nbuckets);
 extern void InitCatCachePhase2(CatCache *cache, bool touch_index);
 
-extern HeapTuple SearchCatCache(CatCache *cache,
+extern HeapTuple (*SearchCatCache)(CatCache *cache,
                                 Datum v1, Datum v2, Datum v3, Datum v4);
-extern HeapTuple SearchCatCache1(CatCache *cache,
+extern HeapTuple (*SearchCatCache1)(CatCache *cache,
                                  Datum v1);
-extern HeapTuple SearchCatCache2(CatCache *cache,
+extern HeapTuple (*SearchCatCache2)(CatCache *cache,
                                  Datum v1, Datum v2);
-extern HeapTuple SearchCatCache3(CatCache *cache,
+extern HeapTuple (*SearchCatCache3)(CatCache *cache,
                                  Datum v1, Datum v2, Datum v3);
-extern HeapTuple SearchCatCache4(CatCache *cache,
+extern HeapTuple (*SearchCatCache4)(CatCache *cache,
                                  Datum v1, Datum v2, Datum v3, Datum v4);
 extern void ReleaseCatCache(HeapTuple tuple);
 
-- 
2.23.0

From bef3df3bb2a0c2340eadf267cdfaf8d40612cd0c Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Date: Fri, 10 Jan 2020 15:08:54 +0900
Subject: [PATCH 4/4] CatCache expiration feature.

This adds the catcache expiration feature to the catcache mechanism.

Current catcache doesn't remove a entry and there's a case where many
hash entries occupy large amont of memory , being not accessed ever
after. This is a quire serious issue on the cases of long-running
sessions.  The expiration feature saves the case in exchange of some
extent of degradation if it is turned on.
---
 src/backend/utils/cache/catcache.c | 343 +++++++++++++++++++++++++++--
 1 file changed, 326 insertions(+), 17 deletions(-)

diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 74c893ba4e..35e1a07e57 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -72,10 +72,11 @@ static CatCacheHeader *CacheHdr = NULL;
 /* Clock for the last accessed time of a catcache entry. */
 TimestampTz    catcacheclock = 0;
 
-static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
-                                               int nkeys,
-                                               Datum v1, Datum v2,
-                                               Datum v3, Datum v4);
+/* basic catcache search functions */
+static inline HeapTuple SearchCatCacheInternalb(CatCache *cache,
+                                                int nkeys,
+                                                Datum v1, Datum v2,
+                                                Datum v3, Datum v4);
 
 static pg_noinline HeapTuple SearchCatCacheMiss(CatCache *cache,
                                                 int nkeys,
@@ -93,6 +94,23 @@ static HeapTuple SearchCatCache3b(CatCache *cache,
 static HeapTuple SearchCatCache4b(CatCache *cache,
                                   Datum v1, Datum v2, Datum v3, Datum v4);
 
+/* catcache search functions with expiration feature */
+static inline HeapTuple SearchCatCacheInternale(CatCache *cache,
+                                                int nkeys,
+                                                Datum v1, Datum v2,
+                                                Datum v3, Datum v4);
+
+static HeapTuple SearchCatCachee(CatCache *cache,
+                                 Datum v1, Datum v2, Datum v3, Datum v4);
+static HeapTuple SearchCatCache1e(CatCache *cache, Datum v1);
+static HeapTuple SearchCatCache2e(CatCache *cache, Datum v1, Datum v2);
+static HeapTuple SearchCatCache3e(CatCache *cache,
+                                  Datum v1, Datum v2, Datum v3);
+static HeapTuple SearchCatCache4e(CatCache *cache,
+                                  Datum v1, Datum v2, Datum v3, Datum v4);
+
+static bool CatCacheCleanupOldEntries(CatCache *cp);
+
 static uint32 CatalogCacheComputeHashValue(CatCache *cache, int nkeys,
                                            Datum v1, Datum v2, Datum v3, Datum v4);
 static uint32 CatalogCacheComputeTupleHashValue(CatCache *cache, int nkeys,
@@ -125,13 +143,35 @@ static SearchCatCacheFuncsType catcache_base = {
     SearchCatCache4b
 };
 
+static SearchCatCacheFuncsType catcache_expire = {
+    SearchCatCachee,
+    SearchCatCache1e,
+    SearchCatCache2e,
+    SearchCatCache3e,
+    SearchCatCache4e
+};
+
 SearchCatCacheFuncsType *SearchCatCacheFuncs = NULL;
 
+/* set catcache function set according to guc variables */
+static void
+set_catcache_functions(void)
+{
+    if (catalog_cache_prune_min_age < 0)
+        SearchCatCacheFuncs = &catcache_base;
+    else
+        SearchCatCacheFuncs = &catcache_expire;
+}
+
+
 /* GUC assign function */
 void
 assign_catalog_cache_prune_min_age(int newval, void *extra)
 {
     catalog_cache_prune_min_age = newval;
+
+    /* choose corresponding function set */
+    set_catcache_functions();
 }
 
 /*
@@ -872,7 +912,7 @@ InitCatCache(int id,
         slist_init(&CacheHdr->ch_caches);
         CacheHdr->ch_ntup = 0;
 
-        SearchCatCacheFuncs = &catcache_base;
+        set_catcache_functions();
 
 #ifdef CATCACHE_STATS
         /* set up to dump stats at backend exit */
@@ -938,6 +978,10 @@ RehashCatCache(CatCache *cp)
     elog(DEBUG1, "rehashing catalog cache id %d for %s; %d tups, %d buckets",
          cp->id, cp->cc_relname, cp->cc_ntup, cp->cc_nbuckets);
 
+    /* try removing old entries before expanding hash */
+    if (CatCacheCleanupOldEntries(cp))
+        return;
+
     /* Allocate a new, larger, hash table. */
     newnbuckets = cp->cc_nbuckets * 2;
     newbucket = (dlist_head *) MemoryContextAllocZero(CacheMemoryContext, newnbuckets * sizeof(dlist_head));
@@ -1222,7 +1266,7 @@ SearchCatCacheb(CatCache *cache,
                Datum v3,
                Datum v4)
 {
-    return SearchCatCacheInternal(cache, cache->cc_nkeys, v1, v2, v3, v4);
+    return SearchCatCacheInternalb(cache, cache->cc_nkeys, v1, v2, v3, v4);
 }
 
 
@@ -1236,7 +1280,7 @@ static HeapTuple
 SearchCatCache1b(CatCache *cache,
                 Datum v1)
 {
-    return SearchCatCacheInternal(cache, 1, v1, 0, 0, 0);
+    return SearchCatCacheInternalb(cache, 1, v1, 0, 0, 0);
 }
 
 
@@ -1244,7 +1288,7 @@ static HeapTuple
 SearchCatCache2b(CatCache *cache,
                 Datum v1, Datum v2)
 {
-    return SearchCatCacheInternal(cache, 2, v1, v2, 0, 0);
+    return SearchCatCacheInternalb(cache, 2, v1, v2, 0, 0);
 }
 
 
@@ -1252,7 +1296,7 @@ static HeapTuple
 SearchCatCache3b(CatCache *cache,
                 Datum v1, Datum v2, Datum v3)
 {
-    return SearchCatCacheInternal(cache, 3, v1, v2, v3, 0);
+    return SearchCatCacheInternalb(cache, 3, v1, v2, v3, 0);
 }
 
 
@@ -1260,19 +1304,19 @@ static HeapTuple
 SearchCatCache4b(CatCache *cache,
                 Datum v1, Datum v2, Datum v3, Datum v4)
 {
-    return SearchCatCacheInternal(cache, 4, v1, v2, v3, v4);
+    return SearchCatCacheInternalb(cache, 4, v1, v2, v3, v4);
 }
 
 /*
- * Work-horse for SearchCatCache/SearchCatCacheN.
+ * Work-horse for SearchCatCacheb/SearchCatCacheNb.
  */
 static inline HeapTuple
-SearchCatCacheInternal(CatCache *cache,
-                       int nkeys,
-                       Datum v1,
-                       Datum v2,
-                       Datum v3,
-                       Datum v4)
+SearchCatCacheInternalb(CatCache *cache,
+                        int nkeys,
+                        Datum v1,
+                        Datum v2,
+                        Datum v3,
+                        Datum v4)
 {
     Datum        arguments[CATCACHE_MAXKEYS];
     uint32        hashValue;
@@ -1497,6 +1541,269 @@ SearchCatCacheMiss(CatCache *cache,
     return &ct->tuple;
 }
 
+/*
+ *    SearchCatCache with entry pruning
+ *
+ *  These functions works the same way with SearchCatCacheNb() functions except
+ *  that less-used entries are removed following catalog_cache_prune_min_age
+ *  setting.
+ */
+static HeapTuple
+SearchCatCachee(CatCache *cache,
+               Datum v1,
+               Datum v2,
+               Datum v3,
+               Datum v4)
+{
+    return SearchCatCacheInternale(cache, cache->cc_nkeys, v1, v2, v3, v4);
+}
+
+
+/*
+ * SearchCatCacheN() are SearchCatCache() versions for a specific number of
+ * arguments. The compiler can inline the body and unroll loops, making them a
+ * bit faster than SearchCatCache().
+ */
+
+static HeapTuple
+SearchCatCache1e(CatCache *cache,
+                Datum v1)
+{
+    return SearchCatCacheInternale(cache, 1, v1, 0, 0, 0);
+}
+
+
+static HeapTuple
+SearchCatCache2e(CatCache *cache,
+                Datum v1, Datum v2)
+{
+    return SearchCatCacheInternale(cache, 2, v1, v2, 0, 0);
+}
+
+
+static HeapTuple
+SearchCatCache3e(CatCache *cache,
+                Datum v1, Datum v2, Datum v3)
+{
+    return SearchCatCacheInternale(cache, 3, v1, v2, v3, 0);
+}
+
+
+static HeapTuple
+SearchCatCache4e(CatCache *cache,
+                Datum v1, Datum v2, Datum v3, Datum v4)
+{
+    return SearchCatCacheInternale(cache, 4, v1, v2, v3, v4);
+}
+
+/*
+ * Work-horse for SearchCatCachee/SearchCatCacheNe.
+ */
+static inline HeapTuple
+SearchCatCacheInternale(CatCache *cache,
+                        int nkeys,
+                        Datum v1,
+                        Datum v2,
+                        Datum v3,
+                        Datum v4)
+{
+    Datum        arguments[CATCACHE_MAXKEYS];
+    uint32        hashValue;
+    Index        hashIndex;
+    dlist_iter    iter;
+    dlist_head *bucket;
+    CatCTup    *ct;
+
+    /* Make sure we're in an xact, even if this ends up being a cache hit */
+    Assert(IsTransactionState());
+
+    Assert(cache->cc_nkeys == nkeys);
+
+    /*
+     * one-time startup overhead for each cache
+     */
+    if (unlikely(cache->cc_tupdesc == NULL))
+        CatalogCacheInitializeCache(cache);
+
+#ifdef CATCACHE_STATS
+    cache->cc_searches++;
+#endif
+
+    /* Initialize local parameter array */
+    arguments[0] = v1;
+    arguments[1] = v2;
+    arguments[2] = v3;
+    arguments[3] = v4;
+
+    /*
+     * find the hash bucket in which to look for the tuple
+     */
+    hashValue = CatalogCacheComputeHashValue(cache, nkeys, v1, v2, v3, v4);
+    hashIndex = HASH_INDEX(hashValue, cache->cc_nbuckets);
+
+    /*
+     * scan the hash bucket until we find a match or exhaust our tuples
+     *
+     * Note: it's okay to use dlist_foreach here, even though we modify the
+     * dlist within the loop, because we don't continue the loop afterwards.
+     */
+    bucket = &cache->cc_bucket[hashIndex];
+    dlist_foreach(iter, bucket)
+    {
+        ct = dlist_container(CatCTup, cache_elem, iter.cur);
+
+        if (ct->dead)
+            continue;            /* ignore dead entries */
+
+        if (ct->hash_value != hashValue)
+            continue;            /* quickly skip entry if wrong hash val */
+
+        if (!CatalogCacheCompareTuple(cache, nkeys, ct->keys, arguments))
+            continue;
+
+        /*
+         * We found a match in the cache.  Move it to the front of the list
+         * for its hashbucket, in order to speed subsequent searches.  (The
+         * most frequently accessed elements in any hashbucket will tend to be
+         * near the front of the hashbucket's list.)
+         */
+        dlist_move_head(bucket, &ct->cache_elem);
+
+        /*
+         * Prolong life of this entry. Since we want run as less instructions
+         * as possible and want the branch be stable for performance reasons,
+         * we don't give a strict cap on the counter. All numbers above 1 will
+         * be regarded as 2 in CatCacheCleanupOldEntries().
+         */
+        ct->naccess++;
+        if (unlikely(ct->naccess == 0))
+            ct->naccess = 2;
+        ct->lastaccess = catcacheclock;
+
+        /*
+         * If it's a positive entry, bump its refcount and return it. If it's
+         * negative, we can report failure to the caller.
+         */
+        if (!ct->negative)
+        {
+            ResourceOwnerEnlargeCatCacheRefs(CurrentResourceOwner);
+            ct->refcount++;
+            ResourceOwnerRememberCatCacheRef(CurrentResourceOwner, &ct->tuple);
+
+            CACHE_elog(DEBUG2, "SearchCatCache(%s): found in bucket %d",
+                       cache->cc_relname, hashIndex);
+
+#ifdef CATCACHE_STATS
+            cache->cc_hits++;
+#endif
+
+            return &ct->tuple;
+        }
+        else
+        {
+            CACHE_elog(DEBUG2, "SearchCatCache(%s): found neg entry in bucket %d",
+                       cache->cc_relname, hashIndex);
+
+#ifdef CATCACHE_STATS
+            cache->cc_neg_hits++;
+#endif
+
+            return NULL;
+        }
+    }
+
+    return SearchCatCacheMiss(cache, nkeys, hashValue, hashIndex, v1, v2, v3, v4);
+}
+
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries happen to be left unused for a long time for several
+ * reasons. Remove such entries to prevent catcache from bloating. It is based
+ * on the similar algorithm with buffer eviction. Entries that are accessed
+ * several times in a certain period live longer than those that have had less
+ * access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+    int        nremoved = 0;
+    int        i;
+    long    oldest_ts = catcacheclock;
+    long    age;
+    int        us;
+
+    /* Return immediately if disabled */
+    if (catalog_cache_prune_min_age < 0)
+        return false;
+
+    /* Don't scan the hash when we know we don't have prunable entries */
+    TimestampDifference(cp->cc_oldest_ts, catcacheclock, &age, &us);
+    if (age < catalog_cache_prune_min_age)
+        return false;
+
+    /* Scan over the whole hash to find entries to remove */
+    for (i = 0 ; i < cp->cc_nbuckets ; i++)
+    {
+        dlist_mutable_iter    iter;
+
+        dlist_foreach_modify(iter, &cp->cc_bucket[i])
+        {
+            CatCTup    *ct = dlist_container(CatCTup, cache_elem, iter.cur);
+
+            /* Don't remove referenced entries */
+            if (ct->refcount == 0 &&
+                (ct->c_list == NULL || ct->c_list->refcount == 0))
+            {
+                /*
+                 * Calculate the duration from the time from the last access
+                 * to the "current" time. catcacheclock is updated
+                 * per-statement basis and additionaly udpated periodically
+                 * during a long running query.
+                 */
+                TimestampDifference(ct->lastaccess, catcacheclock, &age, &us);
+
+                if (age > catalog_cache_prune_min_age)
+                {
+                    /*
+                     * Entries that are not accessed after the last pruning
+                     * are removed in that seconds, and their lives are
+                     * prolonged according to how many times they are accessed
+                     * up to three times of the duration. We don't try shrink
+                     * buckets since pruning effectively caps catcache
+                     * expansion in the long term.
+                     */
+                    if (ct->naccess > 2)
+                        ct->naccess = 1;
+                    else if (ct->naccess > 0)
+                        ct->naccess--;
+                    else
+                    {
+                        CatCacheRemoveCTup(cp, ct);
+                        nremoved++;
+
+                        /* don't update oldest_ts by removed entry */
+                        continue;
+                    }
+                }
+            }
+
+            /* update oldest timestamp if the entry remains alive */
+            if (ct->lastaccess < oldest_ts)
+                oldest_ts = ct->lastaccess;
+        }
+    }
+
+    cp->cc_oldest_ts = oldest_ts;
+
+    if (nremoved > 0)
+        elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+             cp->id, cp->cc_relname, nremoved, cp->cc_ntup + nremoved);
+
+    return nremoved > 0;
+}
+
+
 /*
  *    ReleaseCatCache
  *
@@ -1960,6 +2267,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
     ct->dead = false;
     ct->negative = negative;
     ct->hash_value = hashValue;
+    ct->naccess = 0;
+    ct->lastaccess = catcacheclock;
 
     dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
 
-- 
2.23.0

#! /usr/bin/perl
$collist = "";
foreach $i (0..1000) {
    $collist .= sprintf(", c%05d int", $i);
}
$collist = substr($collist, 2);

printf "drop schema if exists test cascade;\n";
printf "create schema test;\n";
printf "create table test.p ($collist) partition by list (c00000);\n";
foreach $i (0..2999) {
    printf "create table test.t%04d partition of test.p for values in (%d);\n", $i, $i;
}
#!/bin/bash
LOOPS=20
ITERATION=10
BINROOT=/home/horiguti/bin
DATADIR=/home/horiguti/data/data_catexpe
PREC="numeric(10,2)"

/usr/bin/killall postgres
/usr/bin/sleep 3

run() {
    local BINARY=$1
    local PGCTL=$2/bin/pg_ctl
    local PGSQL=$2/bin/postgres
    local PSQL=$2/bin/psql

    if [ "$3" != "" ]; then
      local SETTING1="set catalog_cache_prune_min_age to \"$3\";"
      local SETTING2="set catalog_cache_prune_min_age to \"$4\";"
      local SETTING3="set catalog_cache_prune_min_age to \"$5\";"
    fi

#    ($PGSQL -D $DATADIR 2>&1 > /dev/null)&
    ($PGSQL -D $DATADIR 2>&1 > /dev/null | /usr/bin/sed -e 's/^/# /')&
    /usr/bin/sleep 3
    ${PSQL} postgres <<EOF
create extension if not exists catcachebench;
select catcachebench(0);

$SETTING3

select * from generate_series(2, 2) test,
LATERAL 
  (select '${BINARY}' as version,
          count(r)::text || '/${LOOPS}' as n,
          min(r)::${PREC},
          stddev(r)::${PREC}
   from (select catcachebench(test) as r
            from generate_series(1, ${LOOPS})) r) r

EOF
    $PGCTL --pgdata=$DATADIR stop 2>&1 > /dev/null | /usr/bin/sed -e 's/^/# /'

#    oreport > $BINARY_perf.txt
}

for i in $(seq 0 ${ITERATION}); do
run "master" $BINROOT/pgsql_master_o2 "" "" ""
run "base" $BINROOT/pgsql_catexp-base "" "" ""
run "ind" $BINROOT/pgsql_catexp-ind "" "" ""
run "expire-off" $BINROOT/pgsql_catexpe "-1" "-1" "-1"
run "expire-on" $BINROOT/pgsql_catexpe "300s" "1s" "0"
done

#! /usr/bin/bash

BINROOT=/home/horiguti/bin/pgsql_

for i in master_o2 catexp-base catexp-ind catexpe; do rm -r /home/horiguti/bin/pgsql_$i/*; done
for i in master_o2 catexp-base catexp-ind catexpe; do ls -l /home/horiguti/bin/pgsql_$i/bin/postgres; done


function build () {
    echo $1
    make distclean
    git checkout $2
    git diff master..HEAD > diff_$1.txt
    ./configure --enable-debug --enable-tap-tests --enable-nls --with-openssl --with-libxml --with-llvm
--prefix=${BINROOT}$1LLVM_CONFIG="/usr/bin/llvm-config"
 
    make -sj8 all
    make install
    cd contrib/catcachebench
    make clean
    make all
    make install
    cd ../..
}

build "master_o2" "795e92756cd1"
build "catexp-base" "b2ebc9b4f1c"
build "catexp-ind" "631a04026d"
build "catexpe" "025e5e8a98d"

for i in master_o2 catexp-base catexp-ind catexpe; do ls -l /home/horiguti/bin/pgsql_$i/bin/postgres; done