Re: Protect syscache from bloating with negative cache entries

At Tue, 5 Feb 2019 19:05:26 -0300, Alvaro Herrera <alvherre@2ndquadrant.com> wrote in
<20190205220526.GA1442@alvherre.pgsql>
> On 2019-Feb-05, Tomas Vondra wrote:
> 
> > I don't think we need to remove the expired entries right away, if there
> > are only very few of them. The cleanup requires walking the hash table,
> > which means significant fixed cost. So if there are only few expired
> > entries (say, less than 25% of the cache), we can just leave them around
> > and clean them if we happen to stumble on them (although that may not be
> > possible with dynahash, which has no concept of expiration) of before
> > enlarging the hash table.
> 
> I think seqscanning the hash table is going to be too slow; Ideriha-san
> idea of having a dlist with the entries in LRU order (where each entry
> is moved to head of list when it is touched) seemed good: it allows you
> to evict older ones when the time comes, without having to scan the rest
> of the entries.  Having a dlist means two more pointers on each cache
> entry AFAIR, so it's not a huge amount of memory.

Ah, I had a separate list in my mind. Sounds reasonable to have
pointers in cache entry. But I'm not sure how much additional
dlist_* impact.

The attached is the new version with the following properties:

- Both prune-by-age and hard limiting feature.
  (Merged into single function, single scan)
  Debug tracking feature in CatCacheCleanupOldEntries is removed
  since it no longer runs a full scan.

  Prune-by-age can be a single-setup-for-all-cache feature but
  the hard limit is obviously not. We could use reloptions for
  the purpose (which is not currently available on pg_class and
  pg_attribute:p). I'll add that if there's no strong objection.
  Or is there anyone comes up with something sutable for the
  purpose?

- Using LRU to get rid of full scan.

I added new API dlist_move_to_tail which was needed to construct LRU.

I'm going to retake numbers with search-only queries.

> > So if we want to address this case too (and we probably want), we may
> > need to discard the old cache memory context someho (e.g. rebuild the
> > cache in a new one, and copy the non-expired entries). Which is a nice
> > opportunity to do the "full" cleanup, of course.
> 
> Yeah, we probably don't want to do this super frequently though.

MemoryContext per cache?

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center
From 72a569703662b93fb57c55c337b16107ebccfce3 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horiguchi.kyotaro@lab.ntt.co.jp>
Date: Thu, 7 Feb 2019 14:56:07 +0900
Subject: [PATCH 1/4] Add dlist_move_tail

We have dlist_push_head/tail and dlist_move_head but not
dlist_move_tail. Add it.
---
 src/include/lib/ilist.h | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/src/include/lib/ilist.h b/src/include/lib/ilist.h
index b1a5974ee4..659ab1ac87 100644
--- a/src/include/lib/ilist.h
+++ b/src/include/lib/ilist.h
@@ -394,6 +394,25 @@ dlist_move_head(dlist_head *head, dlist_node *node)
     dlist_check(head);
 }
 
+/*
+ * Move element from its current position in the list to the tail position in
+ * the same list.
+ *
+ * Undefined behaviour if 'node' is not already part of the list.
+ */
+static inline void
+dlist_move_tail(dlist_head *head, dlist_node *node)
+{
+    /* fast path if it's already at the tail */
+    if (head->head.prev == node)
+        return;
+
+    dlist_delete(node);
+    dlist_push_tail(head, node);
+
+    dlist_check(head);
+}
+
 /*
  * Check whether 'node' has a following node.
  * Caution: unreliable if 'node' is not in the list.
-- 
2.16.3

From 5919f1495f27faefdc09abe65fd6e374fa83d9ff Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horiguchi.kyotaro@lab.ntt.co.jp>
Date: Tue, 16 Oct 2018 13:04:30 +0900
Subject: [PATCH 2/4] Remove entries that haven't been used for a certain time

Catcache entries can be left alone for several reasons. It is not
desirable that they eat up memory. With this patch, This adds
consideration of removal of entries that haven't been used for a
certain time before enlarging the hash array.

This also can put a hard limit on the number of catcache entries.
---
 doc/src/sgml/config.sgml                      |  38 ++++++
 src/backend/access/transam/xact.c             |   5 +
 src/backend/utils/cache/catcache.c            | 190 +++++++++++++++++++++++++-
 src/backend/utils/misc/guc.c                  |  63 +++++++++
 src/backend/utils/misc/postgresql.conf.sample |   2 +
 src/include/utils/catcache.h                  |  32 ++++-
 6 files changed, 322 insertions(+), 8 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 9b7a7388d5..d0d2374944 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1662,6 +1662,44 @@ include_dir 'conf.d'
       </listitem>
      </varlistentry>
 
+     <varlistentry id="guc-syscache-memory-target" xreflabel="syscache_memory_target">
+      <term><varname>syscache_memory_target</varname> (<type>integer</type>)
+      <indexterm>
+       <primary><varname>syscache_memory_target</varname> configuration parameter</primary>
+      </indexterm>
+      </term>
+      <listitem>
+       <para>
+        Specifies the maximum amount of memory to which syscache is expanded
+        without pruning. The value defaults to 0, indicating that pruning is
+        always considered. After exceeding this size, syscache pruning is
+        considered according to
+        <xref linkend="guc-syscache-prune-min-age"/>. If you need to keep
+        certain amount of syscache entries with intermittent usage, try
+        increase this setting.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry id="guc-syscache-prune-min-age" xreflabel="syscache_prune_min_age">
+      <term><varname>syscache_prune_min_age</varname> (<type>integer</type>)
+      <indexterm>
+       <primary><varname>syscache_prune_min_age</varname> configuration parameter</primary>
+      </indexterm>
+      </term>
+      <listitem>
+       <para>
+        Specifies the minimum amount of unused time in seconds at which a
+        syscache entry is considered to be removed. -1 indicates that syscache
+        pruning is disabled at all. The value defaults to 600 seconds
+        (<literal>10 minutes</literal>). The syscache entries that are not
+        used for the duration can be removed to prevent syscache bloat. This
+        behavior is suppressed until the size of syscache exceeds
+        <xref linkend="guc-syscache-memory-target"/>.
+       </para>
+      </listitem>
+     </varlistentry>
+
      <varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
       <term><varname>max_stack_depth</varname> (<type>integer</type>)
       <indexterm>
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 92bda87804..ddc433c59e 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -734,7 +734,12 @@ void
 SetCurrentStatementStartTimestamp(void)
 {
     if (!IsParallelWorker())
+    {
         stmtStartTimestamp = GetCurrentTimestamp();
+
+        /* Set this timestamp as aproximated current time */
+        SetCatCacheClock(stmtStartTimestamp);
+    }
     else
         Assert(stmtStartTimestamp != 0);
 }
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 258a1d64cc..0a56390352 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -71,9 +71,32 @@
 #define CACHE6_elog(a,b,c,d,e,f,g)
 #endif
 
+/*
+ * GUC variable to define the minimum size of hash to cosider entry eviction.
+ * This variable is shared among various cache mechanisms.
+ */
+int cache_memory_target = 0;
+
+
+/*
+ * GUC for entry limit. Entries are removed when the number of them goes above
+ * cache_entry_limit by the ratio specified by cache_entry_limit_prune_ratio
+ */
+int cache_entry_limit = 0;
+double cache_entry_limit_prune_ratio = 0.8;
+
+/* GUC variable to define the minimum age of entries that will be cosidered to
+ * be evicted in seconds. This variable is shared among various cache
+ * mechanisms.
+ */
+int cache_prune_min_age = 600;
+
 /* Cache management header --- pointer is NULL until created */
 static CatCacheHeader *CacheHdr = NULL;
 
+/* Timestamp used for any operation on caches. */
+TimestampTz    catcacheclock = 0;
+
 static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
                        int nkeys,
                        Datum v1, Datum v2,
@@ -481,6 +504,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
 
     /* delink from linked list */
     dlist_delete(&ct->cache_elem);
+    dlist_delete(&ct->lru_node);
 
     /*
      * Free keys when we're dealing with a negative entry, normal entries just
@@ -490,6 +514,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
         CatCacheFreeKeys(cache->cc_tupdesc, cache->cc_nkeys,
                          cache->cc_keyno, ct->keys);
 
+    cache->cc_tupsize -= ct->size;
     pfree(ct);
 
     --cache->cc_ntup;
@@ -841,7 +866,9 @@ InitCatCache(int id,
     cp->cc_nkeys = nkeys;
     for (i = 0; i < nkeys; ++i)
         cp->cc_keyno[i] = key[i];
+    cp->cc_tupsize = 0;
 
+    dlist_init(&cp->cc_lru_list);
     /*
      * new cache is initialized as far as we can go for now. print some
      * debugging information, if appropriate.
@@ -858,9 +885,133 @@ InitCatCache(int id,
      */
     MemoryContextSwitchTo(oldcxt);
 
+    /* initilize catcache reference clock if haven't done yet */
+    if (catcacheclock == 0)
+        catcacheclock = GetCurrentTimestamp();
+
     return cp;
 }
 
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries can be left alone for several reasons. We remove them if
+ * they are not accessed for a certain time to prevent catcache from
+ * bloating. The eviction is performed with the similar algorithm with buffer
+ * eviction using access counter. Entries that are accessed several times can
+ * live longer than those that have had no access in the same duration.
+ */
+#define PRUNE_BY_AGE    0x01
+#define PRUNE_BY_NUMBER    0x02
+
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+    int            nremoved = 0;
+    size_t        hash_size;
+    int            nelems_before = cp->cc_ntup;
+    int            ndelelems = 0;
+    int            action = 0;
+    dlist_mutable_iter    iter;
+
+    if (cache_prune_min_age >= 0)
+    {
+        /* prune only if the size of the hash is above the target */
+
+        hash_size = cp->cc_nbuckets * sizeof(dlist_head);
+        if (hash_size + cp->cc_tupsize > (Size) cache_memory_target * 1024L)
+            action |= PRUNE_BY_AGE;
+    }
+
+    if (cache_entry_limit > 0 && nelems_before >= cache_entry_limit)
+    {
+        ndelelems = nelems_before -
+            (int) (cache_entry_limit * cache_entry_limit_prune_ratio);
+
+        if (ndelelems < 256)
+            ndelelems = 256;
+        if (ndelelems > nelems_before)
+            ndelelems = nelems_before;
+
+        action |= PRUNE_BY_NUMBER;
+    }
+
+    /* Return immediately if no pruning is wanted */
+    if (action == 0)
+        return false;
+
+    /* Scan over LRU to find entries to remove */
+    dlist_foreach_modify(iter, &cp->cc_lru_list)
+    {
+        CatCTup    *ct = dlist_container(CatCTup, lru_node, iter.cur);
+        bool        remove_this = false;
+
+        /* We don't remove referenced entry */
+        if (ct->refcount != 0 ||
+            (ct->c_list && ct->c_list->refcount != 0))
+            continue;
+
+        /* check against age */
+        if (action & PRUNE_BY_AGE)
+        {
+            long    entry_age;
+            int        us;
+
+            /*
+             * Calculate the duration from the time of the last access to the
+             * "current" time. Since catcacheclock is not advanced within a
+             * transaction, the entries that are accessed within the current
+             * transaction won't be pruned.
+             */
+            TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+            if (entry_age < cache_prune_min_age)
+            {
+                /* no longer have a business with further entries, exit  */
+                action &= ~PRUNE_BY_AGE;
+                    break;
+            }
+
+            /*
+             * Entries that are not accessed after last pruning are removed in
+             * that seconds, and that has been accessed several times are
+             * removed after leaving alone for up to three times of the
+             * duration. We don't try shrink buckets since pruning effectively
+             * caps catcache expansion in the long term.
+             */
+            if (ct->naccess > 0)
+                ct->naccess--;
+            else 
+                remove_this = true;
+        }
+
+        /* check against entry number */
+        if (action & PRUNE_BY_NUMBER)
+        {
+            if (nremoved < ndelelems)
+                remove_this = true;
+            else
+                action &= ~PRUNE_BY_NUMBER; /* satisfied */
+        }
+
+        /* exit if finished */
+        if (action == 0)
+            break;
+
+        /* do the work */
+        if (remove_this)
+        {
+            CatCacheRemoveCTup(cp, ct);
+            nremoved++;
+        }
+    }
+
+    elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+         cp->id, cp->cc_relname, nremoved, nelems_before);
+
+    return nremoved > 0;
+}
+
 /*
  * Enlarge a catcache, doubling the number of buckets.
  */
@@ -1274,6 +1425,12 @@ SearchCatCacheInternal(CatCache *cache,
          */
         dlist_move_head(bucket, &ct->cache_elem);
 
+        /* Update access information for pruning */
+        if (ct->naccess < 2)
+            ct->naccess++;
+        ct->lastaccess = catcacheclock;
+        dlist_move_tail(&cache->cc_lru_list, &ct->lru_node);
+
         /*
          * If it's a positive entry, bump its refcount and return it. If it's
          * negative, we can report failure to the caller.
@@ -1819,11 +1976,13 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
     CatCTup    *ct;
     HeapTuple    dtp;
     MemoryContext oldcxt;
+    int            tupsize = 0;
 
     /* negative entries have no tuple associated */
     if (ntp)
     {
         int            i;
+        int            tupsize;
 
         Assert(!negative);
 
@@ -1842,13 +2001,14 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
         /* Allocate memory for CatCTup and the cached tuple in one go */
         oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
 
-        ct = (CatCTup *) palloc(sizeof(CatCTup) +
-                                MAXIMUM_ALIGNOF + dtp->t_len);
+        tupsize = sizeof(CatCTup) +    MAXIMUM_ALIGNOF + dtp->t_len;
+        ct = (CatCTup *) palloc(tupsize);
         ct->tuple.t_len = dtp->t_len;
         ct->tuple.t_self = dtp->t_self;
         ct->tuple.t_tableOid = dtp->t_tableOid;
         ct->tuple.t_data = (HeapTupleHeader)
             MAXALIGN(((char *) ct) + sizeof(CatCTup));
+        ct->size = tupsize;
         /* copy tuple contents */
         memcpy((char *) ct->tuple.t_data,
                (const char *) dtp->t_data,
@@ -1876,8 +2036,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
     {
         Assert(negative);
         oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
-        ct = (CatCTup *) palloc(sizeof(CatCTup));
-
+        tupsize = sizeof(CatCTup);
+        ct = (CatCTup *) palloc(tupsize);
         /*
          * Store keys - they'll point into separately allocated memory if not
          * by-value.
@@ -1898,18 +2058,34 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
     ct->dead = false;
     ct->negative = negative;
     ct->hash_value = hashValue;
+    ct->naccess = 0;
+    ct->lastaccess = catcacheclock;
+    dlist_push_tail(&cache->cc_lru_list, &ct->lru_node);
+    ct->size = tupsize;
 
     dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
 
     cache->cc_ntup++;
     CacheHdr->ch_ntup++;
+    cache->cc_tupsize += tupsize;
+
+    /* increase refcount so that this survives pruning */
+    ct->refcount++;
 
     /*
-     * If the hash table has become too full, enlarge the buckets array. Quite
-     * arbitrarily, we enlarge when fill factor > 2.
+     * If the hash table has become too full, try cleanup by removing
+     * infrequently used entries to make a room for the new entry. If it
+     * failed, enlarge the bucket array instead.  Quite arbitrarily, we try
+     * this when fill factor > 2.
      */
-    if (cache->cc_ntup > cache->cc_nbuckets * 2)
+    if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+        !CatCacheCleanupOldEntries(cache))
         RehashCatCache(cache);
+    /* we may still want to prune by entry number, check it */
+    else if (cache_entry_limit > 0 && cache->cc_ntup > cache_entry_limit)
+        CatCacheCleanupOldEntries(cache);
+
+    ct->refcount--;
 
     return ct;
 }
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 8681ada33a..d4df841982 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -81,6 +81,7 @@
 #include "tsearch/ts_cache.h"
 #include "utils/builtins.h"
 #include "utils/bytea.h"
+#include "utils/catcache.h"
 #include "utils/guc_tables.h"
 #include "utils/float.h"
 #include "utils/memutils.h"
@@ -2204,6 +2205,58 @@ static struct config_int ConfigureNamesInt[] =
         NULL, NULL, NULL
     },
 
+    {
+        {"cache_memory_target", PGC_USERSET, RESOURCES_MEM,
+            gettext_noop("Sets the minimum syscache size to keep."),
+            gettext_noop("Cache is not pruned before exceeding this size."),
+            GUC_UNIT_KB
+        },
+        &cache_memory_target,
+        0, 0, MAX_KILOBYTES,
+        NULL, NULL, NULL
+    },
+
+    {
+        {"cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+            gettext_noop("Sets the minimum unused duration of cache entries before removal."),
+            gettext_noop("Cache entries that live unused for longer than this seconds are considered to be
removed."),
+            GUC_UNIT_S
+        },
+        &cache_prune_min_age,
+        600, -1, INT_MAX,
+        NULL, NULL, NULL
+    },
+
+    {
+        {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+            gettext_noop("Sets the maximum entries of catcache."),
+             NULL
+        },
+        &cache_entry_limit,
+        0, 0, INT_MAX,
+        NULL, NULL, NULL
+    },
+
+    {
+        {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+            gettext_noop("Sets the maximum entries of catcache."),
+             NULL
+        },
+        &cache_entry_limit,
+        0, 0, INT_MAX,
+        NULL, NULL, NULL
+    },
+
+    {
+        {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+            gettext_noop("Sets the maximum entries of catcache."),
+             NULL
+        },
+        &cache_entry_limit,
+        0, 0, INT_MAX,
+        NULL, NULL, NULL
+    },
+
     /*
      * We use the hopefully-safely-small value of 100kB as the compiled-in
      * default for max_stack_depth.  InitializeGUCOptions will increase it if
@@ -3368,6 +3421,16 @@ static struct config_real ConfigureNamesReal[] =
         NULL, NULL, NULL
     },
 
+    {
+        {"cache_entry_limit_prune_ratio", PGC_USERSET, RESOURCES_MEM,
+            gettext_noop("Sets the maximum entries of catcache."),
+             NULL
+        },
+        &cache_entry_limit_prune_ratio,
+        0.8, 0.0, 1.0,
+        NULL, NULL, NULL
+    },
+
     /* End-of-list marker */
     {
         {NULL, 0, 0, NULL, NULL}, NULL, 0.0, 0.0, 0.0, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c7f53470df..108d332f2c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,8 @@
 #work_mem = 4MB                # min 64kB
 #maintenance_work_mem = 64MB        # min 1MB
 #autovacuum_work_mem = -1        # min 1MB, or -1 to use maintenance_work_mem
+#cache_memory_target = 0kB    # in kB
+#cache_prune_min_age = 600s    # -1 disables pruning
 #max_stack_depth = 2MB            # min 100kB
 #shared_memory_type = mmap        # the default is the first option
                     # supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..973a87c2cf 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
 
 #include "access/htup.h"
 #include "access/skey.h"
+#include "datatype/timestamp.h"
 #include "lib/ilist.h"
 #include "utils/relcache.h"
 
@@ -61,6 +62,8 @@ typedef struct catcache
     slist_node    cc_next;        /* list link */
     ScanKeyData cc_skey[CATCACHE_MAXKEYS];    /* precomputed key info for heap
                                              * scans */
+    dlist_head    cc_lru_list;
+    int            cc_tupsize;        /* total amount of catcache tuples */
 
     /*
      * Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,7 +122,10 @@ typedef struct catctup
     bool        dead;            /* dead but not yet removed? */
     bool        negative;        /* negative cache entry? */
     HeapTupleData tuple;        /* tuple management header */
-
+    int            naccess;        /* # of access to this entry, up to 2  */
+    TimestampTz    lastaccess;        /* approx. timestamp of the last usage */
+    dlist_node    lru_node;        /* LRU node */
+    int            size;            /* palloc'ed size off this tuple */
     /*
      * The tuple may also be a member of at most one CatCList.  (If a single
      * catcache is list-searched with varying numbers of keys, we may have to
@@ -189,6 +195,30 @@ typedef struct catcacheheader
 /* this extern duplicates utils/memutils.h... */
 extern PGDLLIMPORT MemoryContext CacheMemoryContext;
 
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int cache_prune_min_age;
+extern int cache_memory_target;
+extern int cache_entry_limit;
+extern double cache_entry_limit_prune_ratio;
+
+/* to use as access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/*
+ * SetCatCacheClock - set timestamp for catcache access record
+ */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+    catcacheclock = ts;
+}
+
+static inline TimestampTz
+GetCatCacheClock(void)
+{
+    return catcacheclock;
+}
+
 extern void CreateCacheMemoryContext(void);
 
 extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
-- 
2.16.3

From 2591d5984d5fb9f2fd4cca0ecb8c68431311790a Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi <horiguchi.kyotaro@lab.ntt.co.jp>
Date: Tue, 16 Oct 2018 15:48:28 +0900
Subject: [PATCH 3/4] Syscache usage tracking feature.

Collects syscache usage statictics and show it using the view
pg_stat_syscache. The feature is controlled by the GUC variable
track_syscache_usage_interval.
---
 doc/src/sgml/config.sgml                      |  15 ++
 src/backend/catalog/system_views.sql          |  17 +++
 src/backend/postmaster/pgstat.c               | 201 ++++++++++++++++++++++++--
 src/backend/tcop/postgres.c                   |  23 +++
 src/backend/utils/adt/pgstatfuncs.c           | 134 +++++++++++++++++
 src/backend/utils/cache/catcache.c            |  89 +++++++++---
 src/backend/utils/cache/syscache.c            |  24 +++
 src/backend/utils/init/globals.c              |   1 +
 src/backend/utils/init/postinit.c             |  11 ++
 src/backend/utils/misc/guc.c                  |  10 ++
 src/backend/utils/misc/postgresql.conf.sample |   1 +
 src/include/catalog/pg_proc.dat               |   9 ++
 src/include/miscadmin.h                       |   1 +
 src/include/pgstat.h                          |   6 +-
 src/include/utils/catcache.h                  |   9 +-
 src/include/utils/syscache.h                  |  19 +++
 src/include/utils/timeout.h                   |   1 +
 src/test/regress/expected/rules.out           |  24 ++-
 18 files changed, 559 insertions(+), 36 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d0d2374944..5ff3ebeb4e 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6687,6 +6687,21 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
       </listitem>
      </varlistentry>
 
+     <varlistentry id="guc-track-syscache-usage-interval" xreflabel="track_syscache_usage_interval">
+      <term><varname>track_syscache_usage_interval</varname> (<type>integer</type>)
+      <indexterm>
+       <primary><varname>track_syscache_usage_interval</varname> configuration parameter</primary>
+      </indexterm>
+      </term>
+      <listitem>
+       <para>
+        Specifies the interval to collect system cache usage statistics in
+        milliseconds. This parameter is 0 by default, which means disabled.
+        Only superusers can change this setting.
+       </para>
+      </listitem>
+     </varlistentry>
+
      <varlistentry id="guc-track-io-timing" xreflabel="track_io_timing">
       <term><varname>track_io_timing</varname> (<type>boolean</type>)
       <indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3e229c693c..f5d1aaf96f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -906,6 +906,22 @@ CREATE VIEW pg_stat_progress_vacuum AS
     FROM pg_stat_get_progress_info('VACUUM') AS S
         LEFT JOIN pg_database D ON S.datid = D.oid;
 
+CREATE VIEW pg_stat_syscache AS
+    SELECT
+        S.pid                AS pid,
+        S.relid::regclass    AS relname,
+        S.indid::regclass    AS cache_name,
+        S.size                AS size,
+        S.ntup                AS ntuples,
+        S.searches            AS searches,
+        S.hits                AS hits,
+        S.neg_hits            AS neg_hits,
+        S.ageclass            AS ageclass,
+        S.last_update        AS last_update
+    FROM pg_stat_activity A
+    JOIN LATERAL (SELECT A.pid, * FROM pg_get_syscache_stats(A.pid)) S
+        ON (A.pid = S.pid);
+
 CREATE VIEW pg_user_mappings AS
     SELECT
         U.oid       AS umid,
@@ -1185,6 +1201,7 @@ GRANT EXECUTE ON FUNCTION pg_ls_waldir() TO pg_monitor;
 GRANT EXECUTE ON FUNCTION pg_ls_archive_statusdir() TO pg_monitor;
 GRANT EXECUTE ON FUNCTION pg_ls_tmpdir() TO pg_monitor;
 GRANT EXECUTE ON FUNCTION pg_ls_tmpdir(oid) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_syscache_stats(int) TO pg_monitor;
 
 GRANT pg_read_all_settings TO pg_monitor;
 GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 81c6499251..a1939958b7 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -66,6 +66,7 @@
 #include "utils/ps_status.h"
 #include "utils/rel.h"
 #include "utils/snapmgr.h"
+#include "utils/syscache.h"
 #include "utils/timestamp.h"
 
 
@@ -124,6 +125,7 @@
 bool        pgstat_track_activities = false;
 bool        pgstat_track_counts = false;
 int            pgstat_track_functions = TRACK_FUNC_OFF;
+int            pgstat_track_syscache_usage_interval = 0;
 int            pgstat_track_activity_query_size = 1024;
 
 /* ----------
@@ -236,6 +238,11 @@ typedef struct TwoPhasePgStatRecord
     bool        t_truncated;    /* was the relation truncated? */
 } TwoPhasePgStatRecord;
 
+/* bitmap symbols to specify target file types remove */
+#define PGSTAT_REMFILE_DBSTAT    1        /* remove only databsae stats files */
+#define PGSTAT_REMFILE_SYSCACHE    2        /* remove only syscache stats files */
+#define PGSTAT_REMFILE_ALL        3        /* remove both type of files */
+
 /*
  * Info about current "snapshot" of stats file
  */
@@ -335,6 +342,7 @@ static void pgstat_recv_funcpurge(PgStat_MsgFuncpurge *msg, int len);
 static void pgstat_recv_recoveryconflict(PgStat_MsgRecoveryConflict *msg, int len);
 static void pgstat_recv_deadlock(PgStat_MsgDeadlock *msg, int len);
 static void pgstat_recv_tempfile(PgStat_MsgTempFile *msg, int len);
+static void pgstat_remove_syscache_statsfile(void);
 
 /* ------------------------------------------------------------
  * Public functions called from postmaster follow
@@ -630,10 +638,13 @@ startup_failed:
 }
 
 /*
- * subroutine for pgstat_reset_all
+ * remove stats files
+ *
+ * clean up stats files in specified directory. target is one of
+ * PGSTAT_REFILE_DBSTAT/SYSCACHE/ALL and restricts files to remove.
  */
 static void
-pgstat_reset_remove_files(const char *directory)
+pgstat_reset_remove_files(const char *directory, int target)
 {
     DIR           *dir;
     struct dirent *entry;
@@ -644,25 +655,39 @@ pgstat_reset_remove_files(const char *directory)
     {
         int            nchars;
         Oid            tmp_oid;
+        int            filetype = 0;
 
         /*
          * Skip directory entries that don't match the file names we write.
          * See get_dbstat_filename for the database-specific pattern.
          */
         if (strncmp(entry->d_name, "global.", 7) == 0)
+        {
+            filetype = PGSTAT_REMFILE_DBSTAT;
             nchars = 7;
+        }
         else
         {
+            char head[2];
+            
             nchars = 0;
-            (void) sscanf(entry->d_name, "db_%u.%n",
-                          &tmp_oid, &nchars);
-            if (nchars <= 0)
-                continue;
+            (void) sscanf(entry->d_name, "%c%c_%u.%n",
+                          head, head + 1, &tmp_oid, &nchars);
+
             /* %u allows leading whitespace, so reject that */
-            if (strchr("0123456789", entry->d_name[3]) == NULL)
+            if (nchars < 3 || !isdigit(entry->d_name[3]))
                 continue;
+
+            if  (strncmp(head, "db", 2) == 0)
+                filetype = PGSTAT_REMFILE_DBSTAT;
+            else if (strncmp(head, "cc", 2) == 0)
+                filetype = PGSTAT_REMFILE_SYSCACHE;
         }
 
+        /* skip if this is not a target */
+        if ((filetype & target) == 0)
+            continue;
+
         if (strcmp(entry->d_name + nchars, "tmp") != 0 &&
             strcmp(entry->d_name + nchars, "stat") != 0)
             continue;
@@ -683,8 +708,9 @@ pgstat_reset_remove_files(const char *directory)
 void
 pgstat_reset_all(void)
 {
-    pgstat_reset_remove_files(pgstat_stat_directory);
-    pgstat_reset_remove_files(PGSTAT_STAT_PERMANENT_DIRECTORY);
+    pgstat_reset_remove_files(pgstat_stat_directory, PGSTAT_REMFILE_ALL);
+    pgstat_reset_remove_files(PGSTAT_STAT_PERMANENT_DIRECTORY,
+                              PGSTAT_REMFILE_ALL);
 }
 
 #ifdef EXEC_BACKEND
@@ -2963,6 +2989,10 @@ pgstat_beshutdown_hook(int code, Datum arg)
     if (OidIsValid(MyDatabaseId))
         pgstat_report_stat(true);
 
+    /* clear syscache statistics files and temprary settings */
+    if (MyBackendId != InvalidBackendId)
+        pgstat_remove_syscache_statsfile();
+
     /*
      * Clear my status entry, following the protocol of bumping st_changecount
      * before and after.  We use a volatile pointer here to ensure the
@@ -4287,6 +4317,9 @@ PgstatCollectorMain(int argc, char *argv[])
     pgStatRunningInCollector = true;
     pgStatDBHash = pgstat_read_statsfiles(InvalidOid, true, true);
 
+    /* Remove left-over syscache stats files */
+    pgstat_reset_remove_files(pgstat_stat_directory, PGSTAT_REMFILE_SYSCACHE);
+
     /*
      * Loop to process messages until we get SIGQUIT or detect ungraceful
      * death of our parent postmaster.
@@ -6377,3 +6410,153 @@ pgstat_clip_activity(const char *raw_activity)
 
     return activity;
 }
+
+/*
+ * return the filename for a syscache stat file; filename is the output
+ * buffer, of length len.
+ */
+void
+pgstat_get_syscachestat_filename(bool permanent, bool tempname, int backendid,
+                                 char *filename, int len)
+{
+    int            printed;
+
+    /* NB -- pgstat_reset_remove_files knows about the pattern this uses */
+    printed = snprintf(filename, len, "%s/cc_%u.%s",
+                       permanent ? PGSTAT_STAT_PERMANENT_DIRECTORY :
+                       pgstat_stat_directory,
+                       backendid,
+                       tempname ? "tmp" : "stat");
+    if (printed >= len)
+        elog(ERROR, "overlength pgstat path");
+}
+
+/* removes syscache stats files of this backend */
+static void
+pgstat_remove_syscache_statsfile(void)
+{
+    char    fname[MAXPGPATH];
+
+    pgstat_get_syscachestat_filename(false, false, MyBackendId,
+                                     fname, MAXPGPATH);
+    unlink(fname);        /* don't care of the result */
+}
+
+/*
+ * pgstat_write_syscache_stats() -
+ *        Write the syscache statistics files.
+ *
+ * If 'force' is false, this function skips writing a file and resturns the
+ * time remaining in the current interval in milliseconds. If'force' is true,
+ * writes a file regardless of the remaining time and reset the interval.
+ */
+long
+pgstat_write_syscache_stats(bool force)
+{
+    static TimestampTz last_report = 0;
+    TimestampTz now;
+    long elapsed;
+    long secs;
+    int     usecs;
+    int    cacheId;
+    FILE    *fpout;
+    char    statfile[MAXPGPATH];
+    char    tmpfile[MAXPGPATH];
+
+    /* Return if we don't want it */
+    if (!force && pgstat_track_syscache_usage_interval <= 0)
+    {
+        /* disabled. remove the statistics file if any */
+        if (last_report > 0)
+        {
+            last_report = 0;
+            pgstat_remove_syscache_statsfile();
+        }
+        return 0;
+    }
+    
+    /* Check aginst the interval */
+    now = GetCurrentTransactionStopTimestamp();
+    TimestampDifference(last_report, now, &secs, &usecs);
+    elapsed = secs * 1000 + usecs / 1000;
+
+    if (!force && elapsed < pgstat_track_syscache_usage_interval)
+    {
+        /* not yet the time, inform the remaining time to the caller */
+        return pgstat_track_syscache_usage_interval - elapsed;
+    }
+
+    /* now update the stats */
+    last_report = now;
+
+    pgstat_get_syscachestat_filename(false, true,
+                                     MyBackendId, tmpfile, MAXPGPATH);
+    pgstat_get_syscachestat_filename(false, false,
+                                     MyBackendId, statfile, MAXPGPATH);
+
+    /*
+     * This function can be called from ProcessInterrupts(). Inhibit recursive
+     * interrupts to avoid recursive entry.
+     */
+    HOLD_INTERRUPTS();
+
+    fpout = AllocateFile(tmpfile, PG_BINARY_W);
+    if (fpout == NULL)
+    {
+        ereport(LOG,
+                (errcode_for_file_access(),
+                 errmsg("could not open temporary statistics file \"%s\": %m",
+                        tmpfile)));
+        /*
+         * Failure writing this file is not critical. Just skip this time and
+         * tell caller to wait for the next interval.
+         */
+        RESUME_INTERRUPTS();
+        return pgstat_track_syscache_usage_interval;
+    }
+
+    /* write out every catcache stats */
+    for (cacheId = 0 ; cacheId < SysCacheSize ; cacheId++)
+    {
+        SysCacheStats *stats;
+        
+        stats = SysCacheGetStats(cacheId);
+        Assert (stats);
+
+        /* write error is checked later using ferror() */
+        fputc('T', fpout);
+        (void)fwrite(&cacheId, sizeof(int), 1, fpout);
+        (void)fwrite(&last_report, sizeof(TimestampTz), 1, fpout);
+        (void)fwrite(stats, sizeof(*stats), 1, fpout);
+    }
+    fputc('E', fpout);
+
+    if (ferror(fpout))
+    {
+        ereport(LOG,
+                (errcode_for_file_access(),
+                 errmsg("could not write syscache statistics file \"%s\": %m",
+                        tmpfile)));
+        FreeFile(fpout);
+        unlink(tmpfile);
+    }
+    else if (FreeFile(fpout) < 0)
+    {
+        ereport(LOG,
+                (errcode_for_file_access(),
+                 errmsg("could not close syscache statistics file \"%s\": %m",
+                        tmpfile)));
+        unlink(tmpfile);
+    }
+    else if (rename(tmpfile, statfile) < 0)
+    {
+        ereport(LOG,
+                (errcode_for_file_access(),
+                 errmsg("could not rename syscache statistics file \"%s\" to \"%s\": %m",
+                        tmpfile, statfile)));
+        unlink(tmpfile);
+    }
+
+    RESUME_INTERRUPTS();
+    return 0;
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 36cfd507b2..fb77a0ce4c 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3157,6 +3157,12 @@ ProcessInterrupts(void)
 
     }
 
+    if (IdleSyscacheStatsUpdateTimeoutPending)
+    {
+        IdleSyscacheStatsUpdateTimeoutPending = false;
+        pgstat_write_syscache_stats(true);
+    }
+
     if (ParallelMessagePending)
         HandleParallelMessages();
 }
@@ -3733,6 +3739,7 @@ PostgresMain(int argc, char *argv[],
     sigjmp_buf    local_sigjmp_buf;
     volatile bool send_ready_for_query = true;
     bool        disable_idle_in_transaction_timeout = false;
+    bool        disable_idle_catcache_update_timeout = false;
 
     /* Initialize startup process environment if necessary. */
     if (!IsUnderPostmaster)
@@ -4173,9 +4180,19 @@ PostgresMain(int argc, char *argv[],
             }
             else
             {
+                long timeout;
+
                 ProcessCompletedNotifies();
                 pgstat_report_stat(false);
 
+                timeout = pgstat_write_syscache_stats(false);
+
+                if (timeout > 0)
+                {
+                    disable_idle_catcache_update_timeout = true;
+                    enable_timeout_after(IDLE_CATCACHE_UPDATE_TIMEOUT,
+                                         timeout);
+                }
                 set_ps_display("idle", false);
                 pgstat_report_activity(STATE_IDLE, NULL);
             }
@@ -4218,6 +4235,12 @@ PostgresMain(int argc, char *argv[],
             disable_idle_in_transaction_timeout = false;
         }
 
+        if (disable_idle_catcache_update_timeout)
+        {
+            disable_timeout(IDLE_CATCACHE_UPDATE_TIMEOUT, false);
+            disable_idle_catcache_update_timeout = false;
+        }
+
         /*
          * (6) check for any other interesting events that happened while we
          * slept.
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index b6ba856ebe..6526cfefb4 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -14,6 +14,8 @@
  */
 #include "postgres.h"
 
+#include <sys/stat.h>
+
 #include "access/htup_details.h"
 #include "catalog/pg_authid.h"
 #include "catalog/pg_type.h"
@@ -28,6 +30,7 @@
 #include "utils/acl.h"
 #include "utils/builtins.h"
 #include "utils/inet.h"
+#include "utils/syscache.h"
 #include "utils/timestamp.h"
 
 #define UINT32_ACCESS_ONCE(var)         ((uint32)(*((volatile uint32 *)&(var))))
@@ -1899,3 +1902,134 @@ pg_stat_get_archiver(PG_FUNCTION_ARGS)
     PG_RETURN_DATUM(HeapTupleGetDatum(
                                       heap_form_tuple(tupdesc, values, nulls)));
 }
+
+Datum
+pgstat_get_syscache_stats(PG_FUNCTION_ARGS)
+{
+#define PG_GET_SYSCACHE_SIZE 9
+    int                    pid     = PG_GETARG_INT32(0);
+    ReturnSetInfo       *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+    TupleDesc            tupdesc;
+    Tuplestorestate    *tupstore;
+    MemoryContext        per_query_ctx;
+    MemoryContext        oldcontext;
+    PgBackendStatus       *beentry;
+    int                    beid;
+    char                fname[MAXPGPATH];
+    FILE                  *fpin;
+    char c;
+
+    if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
+        ereport(ERROR,
+                (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+                 errmsg("set-valued function called in context that cannot accept a set")));
+    if (!(rsinfo->allowedModes & SFRM_Materialize))
+        ereport(ERROR,
+                (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+                 errmsg("materialize mode required, but it is not " \
+                        "allowed in this context")));
+
+    /* Build a tuple descriptor for our result type */
+    if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+        elog(ERROR, "return type must be a row type");
+    
+
+    per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;
+
+    oldcontext = MemoryContextSwitchTo(per_query_ctx);
+    tupstore = tuplestore_begin_heap(true, false, work_mem);
+    rsinfo->returnMode = SFRM_Materialize;
+    rsinfo->setResult = tupstore;
+    rsinfo->setDesc = tupdesc;
+
+    MemoryContextSwitchTo(oldcontext);
+
+    /* find beentry for given pid*/
+    beentry = NULL;
+    for (beid = 1;
+         (beentry = pgstat_fetch_stat_beentry(beid)) &&
+             beentry->st_procpid != pid ;
+         beid++);
+
+    /*
+     * we silently return empty result on failure or insufficient privileges
+     */
+    if (!beentry ||
+        (!has_privs_of_role(GetUserId(), beentry->st_userid) &&
+         !is_member_of_role(GetUserId(), DEFAULT_ROLE_READ_ALL_STATS)))
+        goto no_data;
+
+    pgstat_get_syscachestat_filename(false, false, beid, fname, MAXPGPATH);
+
+    if ((fpin = AllocateFile(fname, PG_BINARY_R)) == NULL)
+    {
+        if (errno != ENOENT)
+            ereport(WARNING,
+                    (errcode_for_file_access(),
+                     errmsg("could not open statistics file \"%s\": %m",
+                            fname)));
+        /* also return empty on no statistics file */
+        goto no_data;
+    }
+
+    /* read the statistics file into tuplestore */
+    while ((c = fgetc(fpin)) == 'T')
+    {
+        TimestampTz last_update;
+        SysCacheStats stats;
+        int cacheid;
+        Datum values[PG_GET_SYSCACHE_SIZE];
+        bool nulls[PG_GET_SYSCACHE_SIZE] = {0};
+        Datum datums[SYSCACHE_STATS_NAGECLASSES * 2];
+        bool arrnulls[SYSCACHE_STATS_NAGECLASSES * 2] = {0};
+        int    dims[] = {SYSCACHE_STATS_NAGECLASSES, 2};
+        int lbs[] = {1, 1};
+        ArrayType *arr;
+        int i, j;
+
+        fread(&cacheid, sizeof(int), 1, fpin);
+        fread(&last_update, sizeof(TimestampTz), 1, fpin);
+        if (fread(&stats, 1, sizeof(stats), fpin) != sizeof(stats))
+        {
+            ereport(WARNING, 
+                    (errmsg("corrupted syscache statistics file \"%s\"",
+                            fname)));
+            goto no_data;
+        }
+
+        i = 0;
+        values[i++] = ObjectIdGetDatum(stats.reloid);
+        values[i++] = ObjectIdGetDatum(stats.indoid);
+        values[i++] = Int64GetDatum(stats.size);
+        values[i++] = Int64GetDatum(stats.ntuples);
+        values[i++] = Int64GetDatum(stats.nsearches);
+        values[i++] = Int64GetDatum(stats.nhits);
+        values[i++] = Int64GetDatum(stats.nneg_hits);
+
+        for (j = 0 ; j < SYSCACHE_STATS_NAGECLASSES ; j++)
+        {
+            datums[j * 2] = Int32GetDatum((int32) stats.ageclasses[j]);
+            datums[j * 2 + 1] = Int32GetDatum((int32) stats.nclass_entries[j]);
+        }            
+
+        arr = construct_md_array(datums, arrnulls, 2, dims, lbs,
+                              INT4OID, sizeof(int32), true, 'i');
+        values[i++] = PointerGetDatum(arr);
+
+        values[i++] = TimestampTzGetDatum(last_update);
+
+        Assert (i == PG_GET_SYSCACHE_SIZE);
+
+        tuplestore_putvalues(tupstore, tupdesc, values, nulls);
+    }
+
+    /* check for the end of file. abandon the result if file is broken */
+    if (c != 'E' || fgetc(fpin) != EOF)
+        tuplestore_clear(tupstore);
+
+    FreeFile(fpin);
+
+no_data:
+    tuplestore_donestoring(tupstore);
+    return (Datum) 0;
+}
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 0a56390352..bdcc10064f 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -97,6 +97,10 @@ static CatCacheHeader *CacheHdr = NULL;
 /* Timestamp used for any operation on caches. */
 TimestampTz    catcacheclock = 0;
 
+/* age classes for pruning */
+static double ageclass[SYSCACHE_STATS_NAGECLASSES]
+    = {0.05, 0.1, 1.0, 2.0, 3.0, 0.0};
+
 static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
                        int nkeys,
                        Datum v1, Datum v2,
@@ -628,9 +632,7 @@ CatCacheInvalidate(CatCache *cache, uint32 hashValue)
             else
                 CatCacheRemoveCTup(cache, ct);
             CACHE1_elog(DEBUG2, "CatCacheInvalidate: invalidated");
-#ifdef CATCACHE_STATS
             cache->cc_invals++;
-#endif
             /* could be multiple matches, so keep looking! */
         }
     }
@@ -706,9 +708,7 @@ ResetCatalogCache(CatCache *cache)
             }
             else
                 CatCacheRemoveCTup(cache, ct);
-#ifdef CATCACHE_STATS
             cache->cc_invals++;
-#endif
         }
     }
 }
@@ -958,10 +958,10 @@ CatCacheCleanupOldEntries(CatCache *cp)
             int        us;
 
             /*
-             * Calculate the duration from the time of the last access to the
-             * "current" time. Since catcacheclock is not advanced within a
-             * transaction, the entries that are accessed within the current
-             * transaction won't be pruned.
+             * Calculate the duration from the time from the last access to
+             * the "current" time. Since catcacheclock is not advanced within
+             * a transaction, the entries that are accessed within the current
+             * transaction always get 0 as the result.
              */
             TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
 
@@ -1381,9 +1381,7 @@ SearchCatCacheInternal(CatCache *cache,
     if (unlikely(cache->cc_tupdesc == NULL))
         CatalogCacheInitializeCache(cache);
 
-#ifdef CATCACHE_STATS
     cache->cc_searches++;
-#endif
 
     /* Initialize local parameter array */
     arguments[0] = v1;
@@ -1444,9 +1442,7 @@ SearchCatCacheInternal(CatCache *cache,
             CACHE3_elog(DEBUG2, "SearchCatCache(%s): found in bucket %d",
                         cache->cc_relname, hashIndex);
 
-#ifdef CATCACHE_STATS
             cache->cc_hits++;
-#endif
 
             return &ct->tuple;
         }
@@ -1455,9 +1451,7 @@ SearchCatCacheInternal(CatCache *cache,
             CACHE3_elog(DEBUG2, "SearchCatCache(%s): found neg entry in bucket %d",
                         cache->cc_relname, hashIndex);
 
-#ifdef CATCACHE_STATS
             cache->cc_neg_hits++;
-#endif
 
             return NULL;
         }
@@ -1585,9 +1579,7 @@ SearchCatCacheMiss(CatCache *cache,
     CACHE3_elog(DEBUG2, "SearchCatCache(%s): put in bucket %d",
                 cache->cc_relname, hashIndex);
 
-#ifdef CATCACHE_STATS
     cache->cc_newloads++;
-#endif
 
     return &ct->tuple;
 }
@@ -1698,9 +1690,7 @@ SearchCatCacheList(CatCache *cache,
 
     Assert(nkeys > 0 && nkeys < cache->cc_nkeys);
 
-#ifdef CATCACHE_STATS
     cache->cc_lsearches++;
-#endif
 
     /* Initialize local parameter array */
     arguments[0] = v1;
@@ -1757,9 +1747,7 @@ SearchCatCacheList(CatCache *cache,
         CACHE2_elog(DEBUG2, "SearchCatCacheList(%s): found list",
                     cache->cc_relname);
 
-#ifdef CATCACHE_STATS
         cache->cc_lhits++;
-#endif
 
         return cl;
     }
@@ -2276,3 +2264,64 @@ PrintCatCacheListLeakWarning(CatCList *list)
          list->my_cache->cc_relname, list->my_cache->id,
          list, list->refcount);
 }
+
+/*
+ * CatCacheGetStats - fill in SysCacheStats struct.
+ *
+ * This is a support routine for SysCacheGetStats, substantially fills in the
+ * result. The classification here is based on the same criteria to
+ * CatCacheCleanupOldEntries().
+ */
+void
+CatCacheGetStats(CatCache *cache, SysCacheStats *stats)
+{
+    int    i, j;
+
+    Assert(ageclass[SYSCACHE_STATS_NAGECLASSES - 1] == 0.0);
+
+    /* fill in the stats struct */
+    stats->size = cache->cc_tupsize + cache->cc_nbuckets * sizeof(dlist_head);
+    stats->ntuples = cache->cc_ntup;
+    stats->nsearches = cache->cc_searches;
+    stats->nhits = cache->cc_hits;
+    stats->nneg_hits = cache->cc_neg_hits;
+
+    /* cache_prune_min_age can be changed on-session, fill it every time */
+    for (i = 0 ; i < SYSCACHE_STATS_NAGECLASSES ; i++)
+        stats->ageclasses[i] = (int) (cache_prune_min_age * ageclass[i]);
+
+    /*
+     * nth element in nclass_entries stores the number of cache entries that
+     * have lived unaccessed for corresponding multiple in ageclass of
+     * cache_prune_min_age.
+     */
+    memset(stats->nclass_entries, 0, sizeof(int) * SYSCACHE_STATS_NAGECLASSES);
+
+    /* Scan the whole hash */
+    for (i = 0; i < cache->cc_nbuckets; i++)
+    {
+        dlist_mutable_iter iter;
+
+        dlist_foreach_modify(iter, &cache->cc_bucket[i])
+        {
+            CatCTup    *ct = dlist_container(CatCTup, cache_elem, iter.cur);
+            long entry_age;
+            int us;
+
+            /*
+             * Calculate the duration from the time from the last access to
+             * the "current" time. Since catcacheclock is not advanced within
+             * a transaction, the entries that are accessed within the current
+             * transaction won't be pruned.
+             */
+            TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+            j = 0;
+            while (j < SYSCACHE_STATS_NAGECLASSES - 1 &&
+                   entry_age > stats->ageclasses[j])
+                j++;
+
+            stats->nclass_entries[j]++;
+        }
+    }
+}
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index ac98c19155..7b38a06708 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -20,6 +20,9 @@
  */
 #include "postgres.h"
 
+#include <sys/stat.h>
+#include <unistd.h>
+
 #include "access/htup_details.h"
 #include "access/sysattr.h"
 #include "catalog/indexing.h"
@@ -1534,6 +1537,27 @@ RelationSupportsSysCache(Oid relid)
     return false;
 }
 
+/*
+ * SysCacheGetStats - returns stats of specified syscache
+ *
+ * This routine returns the address of its local static memory.
+ */
+SysCacheStats *
+SysCacheGetStats(int cacheId)
+{
+    static SysCacheStats stats;
+
+    Assert(cacheId >=0 && cacheId < SysCacheSize);
+
+    memset(&stats, 0, sizeof(stats));
+
+    stats.reloid = cacheinfo[cacheId].reloid;
+    stats.indoid = cacheinfo[cacheId].indoid;
+
+    CatCacheGetStats(SysCache[cacheId], &stats);
+
+    return &stats;
+}
 
 /*
  * OID comparator for pg_qsort
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index fd51934aaf..f039ecd805 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -32,6 +32,7 @@ volatile sig_atomic_t QueryCancelPending = false;
 volatile sig_atomic_t ProcDiePending = false;
 volatile sig_atomic_t ClientConnectionLost = false;
 volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
+volatile sig_atomic_t IdleSyscacheStatsUpdateTimeoutPending = false;
 volatile sig_atomic_t ConfigReloadPending = false;
 volatile uint32 InterruptHoldoffCount = 0;
 volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index c0b6231458..dee7f19475 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -72,6 +72,7 @@ static void ShutdownPostgres(int code, Datum arg);
 static void StatementTimeoutHandler(void);
 static void LockTimeoutHandler(void);
 static void IdleInTransactionSessionTimeoutHandler(void);
+static void IdleSyscacheStatsUpdateTimeoutHandler(void);
 static bool ThereIsAtLeastOneRole(void);
 static void process_startup_options(Port *port, bool am_superuser);
 static void process_settings(Oid databaseid, Oid roleid);
@@ -628,6 +629,8 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
         RegisterTimeout(LOCK_TIMEOUT, LockTimeoutHandler);
         RegisterTimeout(IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
                         IdleInTransactionSessionTimeoutHandler);
+        RegisterTimeout(IDLE_CATCACHE_UPDATE_TIMEOUT,
+                        IdleSyscacheStatsUpdateTimeoutHandler);
     }
 
     /*
@@ -1239,6 +1242,14 @@ IdleInTransactionSessionTimeoutHandler(void)
     SetLatch(MyLatch);
 }
 
+static void
+IdleSyscacheStatsUpdateTimeoutHandler(void)
+{
+    IdleSyscacheStatsUpdateTimeoutPending = true;
+    InterruptPending = true;
+    SetLatch(MyLatch);
+}
+
 /*
  * Returns true if at least one role is defined in this database cluster.
  */
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index d4df841982..7bb239a07e 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -3198,6 +3198,16 @@ static struct config_int ConfigureNamesInt[] =
         NULL, NULL, NULL
     },
 
+    {
+        {"track_syscache_usage_interval", PGC_SUSET, STATS_COLLECTOR,
+            gettext_noop("Sets the interval between syscache usage collection, in milliseconds. Zero disables syscache
usagetracking."),
 
+            NULL
+        },
+        &pgstat_track_syscache_usage_interval,
+        0, 0, INT_MAX / 2,
+        NULL, NULL, NULL
+    },
+
     {
         {"gin_pending_list_limit", PGC_USERSET, CLIENT_CONN_STATEMENT,
             gettext_noop("Sets the maximum size of the pending list for GIN index."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 108d332f2c..4d4fb42251 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -560,6 +560,7 @@
 #track_io_timing = off
 #track_functions = none            # none, pl, all
 #track_activity_query_size = 1024    # (change requires restart)
+#track_syscache_usage_interval = 0    # zero disables tracking
 #stats_temp_directory = 'pg_stat_tmp'
 
 
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index b8de13f03b..6099a828d2 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9669,6 +9669,15 @@
   proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
   proargnames =>
'{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn}',
   prosrc => 'pg_get_replication_slots' },
+{ oid => '3425',
+  descr => 'syscache statistics',
+  proname => 'pg_get_syscache_stats', prorows => '100', proisstrict => 'f',
+  proretset => 't', provolatile => 'v', prorettype => 'record',
+  proargtypes => 'int4',
+  proallargtypes => '{int4,oid,oid,int8,int8,int8,int8,int8,_int4,timestamptz}',
+  proargmodes => '{i,o,o,o,o,o,o,o,o,o}',
+  proargnames => '{pid,relid,indid,size,ntup,searches,hits,neg_hits,ageclass,last_update}',
+  prosrc => 'pgstat_get_syscache_stats' },
 { oid => '3786', descr => 'set up a logical replication slot',
   proname => 'pg_create_logical_replication_slot', provolatile => 'v',
   proparallel => 'u', prorettype => 'record', proargtypes => 'name name bool',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index c9e35003a5..69b9a976f0 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -82,6 +82,7 @@ extern PGDLLIMPORT volatile sig_atomic_t InterruptPending;
 extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
 extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
 extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t IdleSyscacheStatsUpdateTimeoutPending;
 extern PGDLLIMPORT volatile sig_atomic_t ConfigReloadPending;
 
 extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 88a75fb798..b6bfd7d644 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -1144,6 +1144,7 @@ extern bool pgstat_track_activities;
 extern bool pgstat_track_counts;
 extern int    pgstat_track_functions;
 extern PGDLLIMPORT int pgstat_track_activity_query_size;
+extern int    pgstat_track_syscache_usage_interval;
 extern char *pgstat_stat_directory;
 extern char *pgstat_stat_tmpname;
 extern char *pgstat_stat_filename;
@@ -1228,7 +1229,8 @@ extern PgStat_BackendFunctionEntry *find_funcstat_entry(Oid func_id);
 extern void pgstat_initstats(Relation rel);
 
 extern char *pgstat_clip_activity(const char *raw_activity);
-
+extern void pgstat_get_syscachestat_filename(bool permanent,
+                    bool tempname, int backendid, char *filename, int len);
 /* ----------
  * pgstat_report_wait_start() -
  *
@@ -1363,5 +1365,5 @@ extern PgStat_StatFuncEntry *pgstat_fetch_stat_funcentry(Oid funcid);
 extern int    pgstat_fetch_stat_numbackends(void);
 extern PgStat_ArchiverStats *pgstat_fetch_stat_archiver(void);
 extern PgStat_GlobalStats *pgstat_fetch_global(void);
-
+extern long pgstat_write_syscache_stats(bool force);
 #endif                            /* PGSTAT_H */
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 973a87c2cf..85fa7bdb86 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -66,10 +66,8 @@ typedef struct catcache
     int            cc_tupsize;        /* total amount of catcache tuples */
 
     /*
-     * Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
-     * doesn't break ABI for other modules
+     * Statistics entries
      */
-#ifdef CATCACHE_STATS
     long        cc_searches;    /* total # searches against this cache */
     long        cc_hits;        /* # of matches against existing entry */
     long        cc_neg_hits;    /* # of matches against negative entry */
@@ -82,7 +80,6 @@ typedef struct catcache
     long        cc_invals;        /* # of entries invalidated from cache */
     long        cc_lsearches;    /* total # list-searches */
     long        cc_lhits;        /* # of matches against existing lists */
-#endif
 } CatCache;
 
 
@@ -258,4 +255,8 @@ extern void PrepareToInvalidateCacheTuple(Relation relation,
 extern void PrintCatCacheLeakWarning(HeapTuple tuple);
 extern void PrintCatCacheListLeakWarning(CatCList *list);
 
+/* defined in syscache.h */
+typedef struct syscachestats SysCacheStats;
+extern void CatCacheGetStats(CatCache *cache, SysCacheStats *syscachestats);
+
 #endif                            /* CATCACHE_H */
diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h
index 95ee48954e..71b399c902 100644
--- a/src/include/utils/syscache.h
+++ b/src/include/utils/syscache.h
@@ -112,6 +112,24 @@ enum SysCacheIdentifier
 #define SysCacheSize (USERMAPPINGUSERSERVER + 1)
 };
 
+#define SYSCACHE_STATS_NAGECLASSES 6
+/* Struct for catcache tracking information */
+typedef struct syscachestats
+{
+    Oid        reloid;            /* target relation */
+    Oid        indoid;            /* index */
+    size_t    size;            /* size of the catcache */
+    int        ntuples;        /* number of tuples resides in the catcache */
+    int        nsearches;        /* number of searches */
+    int        nhits;            /* number of cache hits */
+    int        nneg_hits;        /* number of negative cache hits */
+    /* age classes in seconds */
+    int        ageclasses[SYSCACHE_STATS_NAGECLASSES];
+    /* number of tuples fall into the corresponding age class */
+    int        nclass_entries[SYSCACHE_STATS_NAGECLASSES];
+} SysCacheStats;
+
+
 extern void InitCatalogCache(void);
 extern void InitCatalogCachePhase2(void);
 
@@ -164,6 +182,7 @@ extern void SysCacheInvalidate(int cacheId, uint32 hashValue);
 extern bool RelationInvalidatesSnapshotsOnly(Oid relid);
 extern bool RelationHasSysCache(Oid relid);
 extern bool RelationSupportsSysCache(Oid relid);
+extern SysCacheStats *SysCacheGetStats(int cacheId);
 
 /*
  * The use of the macros below rather than direct calls to the corresponding
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 9244a2a7b7..0ab441a364 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -31,6 +31,7 @@ typedef enum TimeoutId
     STANDBY_TIMEOUT,
     STANDBY_LOCK_TIMEOUT,
     IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
+    IDLE_CATCACHE_UPDATE_TIMEOUT,
     /* First user-definable timeout reason */
     USER_TIMEOUT,
     /* Maximum number of timeout reasons */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2c8e21baa7..7bd77e9972 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1921,6 +1921,28 @@ pg_stat_sys_tables| SELECT pg_stat_all_tables.relid,
     pg_stat_all_tables.autoanalyze_count
    FROM pg_stat_all_tables
   WHERE ((pg_stat_all_tables.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR
(pg_stat_all_tables.schemaname~ '^pg_toast'::text));
 
+pg_stat_syscache| SELECT s.pid,
+    (s.relid)::regclass AS relname,
+    (s.indid)::regclass AS cache_name,
+    s.size,
+    s.ntup AS ntuples,
+    s.searches,
+    s.hits,
+    s.neg_hits,
+    s.ageclass,
+    s.last_update
+   FROM (pg_stat_activity a
+     JOIN LATERAL ( SELECT a.pid,
+            pg_get_syscache_stats.relid,
+            pg_get_syscache_stats.indid,
+            pg_get_syscache_stats.size,
+            pg_get_syscache_stats.ntup,
+            pg_get_syscache_stats.searches,
+            pg_get_syscache_stats.hits,
+            pg_get_syscache_stats.neg_hits,
+            pg_get_syscache_stats.ageclass,
+            pg_get_syscache_stats.last_update
+           FROM pg_get_syscache_stats(a.pid) pg_get_syscache_stats(relid, indid, size, ntup, searches, hits, neg_hits,
ageclass,last_update)) s ON ((a.pid = s.pid)));
 
 pg_stat_user_functions| SELECT p.oid AS funcid,
     n.nspname AS schemaname,
     p.proname AS funcname,
@@ -2352,7 +2374,7 @@ pg_settings|pg_settings_n|CREATE RULE pg_settings_n AS
     ON UPDATE TO pg_catalog.pg_settings DO INSTEAD NOTHING;
 pg_settings|pg_settings_u|CREATE RULE pg_settings_u AS
     ON UPDATE TO pg_catalog.pg_settings
-   WHERE (new.name = old.name) DO  SELECT set_config(old.name, new.setting, false) AS set_config;
+   WHERE (new.name = old.name) DO  SELECT set_config(old.name, new.setting, false, false) AS set_config;
 rtest_emp|rtest_emp_del|CREATE RULE rtest_emp_del AS
     ON DELETE TO public.rtest_emp DO  INSERT INTO rtest_emplog (ename, who, action, newsal, oldsal)
   VALUES (old.ename, CURRENT_USER, 'fired'::bpchar, '$0.00'::money, old.salary);
-- 
2.16.3