Everyone,
I’ve run into an issue that’s got me stumped and I would be really grateful for any ideas. We’re deploying Postgres 11.8 on Kubernetes 1.17.9. The nodes are running RedHat EL 7.9 with the latest kernel for that distribution.
We create a Kubernetes pod that runs Postgres. It’s setup with /pgdata going to a PersistentVolumeClaim that’s thin provisioned with a size set at 500GB. Initially, the allocated size of the PersistentVolumeClaim’s corresponding disk file (VMware .VMDK) is around 12GB. However, the size of the .VMDK keeps increasing at roughly 150-300MB/Hour even though the size of the /pgdata folder as shown by df is stable around 230MB. To try to troubleshoot this, I set up Postgres to put the pg_wal folder on a different partition. That didn’t make any difference.
Replication is turned on for this Postgres instance, and it was observed that the replica’s .VMDK file grew at a rate of 25MB/Hour.
I’ve checked to make sure that the partition is mounted with the DISCARD option. I created a test program that would allocate disk space, and then free it. I confirmed that when DISCARD is used for mount, VMware will reclaim the space as expected. For example, if the .VMDK grows to 20GB during a test, it shrinks back down to some size like 1GB when the test completes.
To try to sort this out, I’ve done the following:
- Written the test program and confirmed that on the ESXI Host/VCenter space reclamation works.
- Moved the pg_wal directory to another partition.
- Changed Auto-Vacuum frequency to run less often (12 hours).
- Executed “vacuum full”, followed by fstrim </mount point> from the Kubernetes node. This freed perhaps 12 MB from the /pgdata folder, and around 19MB from the .VMDK size.
- Checked the /proc/fd descriptors for the running connections and confirmed no unusual temp files are opened.
- Confirmed the replication is working as expected.
- Set archive_mode off to eliminate that as a source of noise.
Does anyone have any ideas about what’s causing this? Is there anything unusual about how Postgres allocates temporary data files or frees them? I’m really just grasping. Thanks for looking!
George