Diagnose memory leak in logical replication?

Dear all,

Hope everyone is doing well.

While playing with wal2json and debezium we faced an issue where 
walsender processes eat more and more memory (several hundred megabytes 
per hour) until we restart them by stopping their consumption.

We first thought to something wrong in wal2json[1] but after more 
testing today we have the exact same situation with pgoutput plugin.

The environment is Debian buster with packages from apt.postgresql.org 
(currently using 12.2-2.pgdg100+1):

postgresql-12 - 12.2-2.pgdg100+1
postgresql-12-wal2json - 2.2-1.pgdg100+1
postgresql-client-12 - 12.2-2.pgdg100+1
postgresql-client-common - 213.pgdg100+1
postgresql-common - 213.pgdg100+1

The steps to create slot and consume wal2json are the following:

pg_recvlogical -d test_db -U test_u -p 5433 -W --slot test_slot 
--create-slot -P wal2json
pg_recvlogical -d test_db -U test_u -p 5433 -W --slot test_slot --start 
-f /dev/null

The steps done to create slots and consume with pgoutput plugin are the 
following:
In PostgreSQL:
CREATE PUBLICATION dbz_minimal_publication FOR TABLE public.xxx;

And then from the shell:
pg_recvlogical -d test_db --slot pgoutput_minimal_test_slot 
--create-slot -P pgoutput
pg_recvlogical -d test_db -U test_u -p 5433 -W --slot 
pgoutput_minimal_test_slot --start -h localhost -o proto_version=1 -o 
publication_names='dbz_minimal_publication' -f /dev/null


A few minutes after creating these slots (Thu Nov  5 12:55:45 2020)
Every 5.0s: ps eo user,pid,vsz,rss $(pgrep -f walsender)
USER       PID    VSZ   RSS
postgres  3080 11672352 916664  <- this is the walsender using wal2json 
plugin
postgres  5740 11533412 760332 <- this is the walsender using pgoutput 
plugin
...
postgres 14333 13011864 2246328 <- those are other walsender using 
wal2json plugin
postgres 14383 13011864 2244788
postgres 14384 13011864 2246316

Same thing a few hours later: Thu Nov  5 14:01:55 2020
Every 5.0s: ps eo user,pid,vsz,rss $(pgrep -f walsender)
USER       PID    VSZ   RSS
postgres  3080 12153832 1400920 <- this is the walsender using wal2json 
plugin
postgres  5740 12014716 1245524 <- this is the walsender using pgoutput 
plugin
...
postgres 14333 13493032 2728520
postgres 14383 13493032 2727012
postgres 14384 13493032 2728672

Per wal2json author suggestion and based on other posts on this list we 
attached gdb to one of the walsender process a few weeks ago and did the 
following:
p (long)MemoryContextStats((long)TopMemoryContext)

It seems we have a lot of ReorderBufferToastHash under the Logical 
decoding context:
TopMemoryContext: 435000 total in 9 blocks; 29864 free (17 chunks); 
405136 used
   Type information cache: 24352 total in 2 blocks; 2624 free (0 chunks); 
21728 used
   pgstat TabStatusArray lookup hash table: 8192 total in 1 blocks; 1456 
free (0 chunks); 6736 used
   wal2json output context: 8192 total in 1 blocks; 7936 free (0 chunks); 
256 used
   RowDescriptionContext: 8192 total in 1 blocks; 6896 free (0 chunks); 
1296 used
   MessageContext: 8192 total in 1 blocks; 6896 free (1 chunks); 1296 
used
     Replication command context: 82104 total in 5 blocks; 25848 free (12 
chunks); 56256 used
       Logical decoding context: 344176 total in 5 blocks; 26672 free (12 
chunks); 317504 used
         snapshot builder context: 524288 total in 7 blocks; 172928 free 
(24 chunks); 351360 used
         ReorderBuffer: 32768 total in 3 blocks; 28048 free (26 chunks); 
4720 used
           ReorderBufferToastHash: 524288 total in 7 blocks; 28784 free 
(22 chunks); 495504 used
           ReorderBufferToastHash: 524288 total in 7 blocks; 28784 free 
(22 chunks); 495504 used
           ReorderBufferToastHash: 524288 total in 7 blocks; 28784 free 
(22 chunks); 495504 used
           ReorderBufferToastHash: 524288 total in 7 blocks; 28784 free 
(22 chunks); 495504 used
...
        ReorderBufferByXid: 16384 total in 2 blocks; 3544 free (2 
chunks); 12840 used
           Tuples: 2080374888 total in 248 blocks (1259603 chunks); 
241400 free (265927 chunks); 2080133488 used
           TXN: 648 total in 0 blocks; 0 free (0 chunks); 648 used
           Change: 79881960 total in 9751 blocks; 74080 free (926 
chunks); 79807880 used
   Operator class cache: 8192 total in 1 blocks; 560 free (0 chunks); 
7632 used

The full output can be found on github[2]

Please note that all of this seems to be related to the DB activity, 
simply doing a test with for example a pgbench in loop does not allow to 
reproduce this "leak".

Would someone point me in the right direction to identify what could be 
the reason for this behaviour so that we can try to fix this?

Thanks,
Raph
[1] https://github.com/eulerto/wal2json/issues/180
[2] https://github.com/eulerto/wal2json/files/5292603/memory.txt