FATAL: terminating walreceiver process due to administrator command

Hello,

The application is monitoring the delay. In the past, it happened a few times I received this notification during a high load but not from this server.

However, the replication was not lagging. It stopped.

There was a single line in the log of the standby server:

FATAL:  terminating walreceiver process due to administrator command

I did not find anything related in the master’s log. The rest of the log was all about the slow statements.

As far I can tell you, this never happened before. The standby server was running but the replication stopped. I restarted the server after 42 minutes of that line in the log. The replication caught up in 3-5 seconds.

Only I have access to the servers.

I did not stop the replication process. I don’t even know how to do it.

There is no cron task that would do such thing. Only one application access the database. I wrote it hence I know it didn’t do it either.

I have been running the servers for years with more or less the same configuration.

As far as I see, when I see the same line in earlier logs, the database was shut down as well. This was the only lonely line like that.

Recent changes on the servers:

* On 11 January, I upgraded from 10.5 to 10.6_2

* A few days ago, set up a new server that replicates one table from the same master. This is a huge table but it’s rarely written. The replication works. It’s the same time I used logical replication. The server where the replication stopped uses async stream replication.

* When I set up the logical replication, I increased the wal_sender_timeout 

I found nothing related in the logs (/var/log/messages, /var/log/all.log, dmesg). This slave is probably the least loaded server of the group.

FreeBSD xxx 11.2-RELEASE-p8 FreeBSD 11.2-RELEASE-p8 #0: Tue Jan  8 21:35:12 UTC 2019     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

/boot/loader.conf:

Everything else is either FreeBSD default or unrelated.

There is a lot of free memory. I don’t mean usable but free. 3GB RAM was not even touched since the last boot.

M.