We ran into a production issue during our maintenance window when I switched over LifeKeeper/PostgreSQL nodes from one to another. We noticed that the database was 1000 times slower than it usually is, then realized that the log_min_messages setting was set to 'debug5' for no reason.
Facts:
- The postgresql.conf file has always been using the default value, which is 'warning'.
- An attempt to explicitly set log_min_messages=warning and reload could not override the setting of 'debug5'.
- The heavy debug5 logging on our production system drastically slowed down our database, and eventually filled up the root file system (due to the huge /var/log/postgresql)
- The LifeKeeper failover has been done in a recent past with the same unmodified postgresql.log file without such an excessive logging behavior.
- Changing log_destination from syslog to stderr appeared to have done the trick, after hours of troubleshooting. The log_min_messages value took the explicit setting from postgresql.log ('warning') and restored our database performance.
I could not find any relevant issue reported and want to see if anyone has a clue about this issue.
Thank you,
-Kong