Size of /pgdata
Size of /pgdata
Re: Size of /pgdata
Hi all,I am using 17.9 version.I see suddenly du -sh /pgdata/17 shows 930GB. Total allocated is 1TB.I checked the db size using \l+ and also pg_total_relation_size and size for particular db is 630GB. Ignoring postgres and template dbs as they are in KBs.Now where the remaining 300 comes from approximately..?I checked pg_wal and is 100gb..that is intended based on max_wal_size setting.Postgres logs also in MBs ..so ruling out that.Now 300 - 100 = 200gb approximately don't know where it comes from.So du -sh /base is approximately 830Gb whereas actual db size is 630gb as mentioned earlier.I took one big sample table which is 73GB, and using relfilenode(327683) I tried finding the underlying file size du -sh 327683* and able to find the total size the files occpuing and found 113gb.......so for one table itself it is 40Gb additional...I also checked the bloat of table using pgstatstuple and no bloat observed for this sample tableWhy we have so much increase of file size? How we can avoid or tune it? Is it about reallocating files and not being used fully? What's the solution?
Re: Size of /pgdata
Probably, pgsql_tmp file might consumed rest of ~200gb disk. You will find this inside base directory.During large query executions, if data to be sorted exceeds work_mem settings, postgresql creates temporary files and spills data to disk.Possible reasons :Poor queryLarge batch jobsMissing indexLRQ’sToo low work_memOn Sat, 13 Jun 2026 at 12:18 PM, Raj <rajeshkumar.dba09@gmail.com> wrote:Hi all,I am using 17.9 version.I see suddenly du -sh /pgdata/17 shows 930GB. Total allocated is 1TB.I checked the db size using \l+ and also pg_total_relation_size and size for particular db is 630GB. Ignoring postgres and template dbs as they are in KBs.Now where the remaining 300 comes from approximately..?I checked pg_wal and is 100gb..that is intended based on max_wal_size setting.Postgres logs also in MBs ..so ruling out that.Now 300 - 100 = 200gb approximately don't know where it comes from.So du -sh /base is approximately 830Gb whereas actual db size is 630gb as mentioned earlier.I took one big sample table which is 73GB, and using relfilenode(327683) I tried finding the underlying file size du -sh 327683* and able to find the total size the files occpuing and found 113gb.......so for one table itself it is 40Gb additional...I also checked the bloat of table using pgstatstuple and no bloat observed for this sample tableWhy we have so much increase of file size? How we can avoid or tune it? Is it about reallocating files and not being used fully? What's the solution?
Re: Size of /pgdata
Checked temp files as well...no thesee are not temp.filesOn Sat, 13 Jun 2026, 12:39 Bhanu Prakash, <mallelabhanuprakash@gmail.com> wrote:Probably, pgsql_tmp file might consumed rest of ~200gb disk. You will find this inside base directory.During large query executions, if data to be sorted exceeds work_mem settings, postgresql creates temporary files and spills data to disk.Possible reasons :Poor queryLarge batch jobsMissing indexLRQ’sToo low work_memOn Sat, 13 Jun 2026 at 12:18 PM, Raj <rajeshkumar.dba09@gmail.com> wrote:Hi all,I am using 17.9 version.I see suddenly du -sh /pgdata/17 shows 930GB. Total allocated is 1TB.I checked the db size using \l+ and also pg_total_relation_size and size for particular db is 630GB. Ignoring postgres and template dbs as they are in KBs.Now where the remaining 300 comes from approximately..?I checked pg_wal and is 100gb..that is intended based on max_wal_size setting.Postgres logs also in MBs ..so ruling out that.Now 300 - 100 = 200gb approximately don't know where it comes from.So du -sh /base is approximately 830Gb whereas actual db size is 630gb as mentioned earlier.I took one big sample table which is 73GB, and using relfilenode(327683) I tried finding the underlying file size du -sh 327683* and able to find the total size the files occpuing and found 113gb.......so for one table itself it is 40Gb additional...I also checked the bloat of table using pgstatstuple and no bloat observed for this sample tableWhy we have so much increase of file size? How we can avoid or tune it? Is it about reallocating files and not being used fully? What's the solution?
Re: Size of /pgdata
On Sat, 2026-06-13 at 12:18 +0530, Raj wrote:
> I took one big sample table which is 73GB, and using relfilenode(327683) I tried finding the
> underlying file size du -sh 327683* and able to find the total size the files occpuing and
> found 113gb.......so for one table itself it is 40Gb additional...
Investigate more:
- Make sure that you don't get files from another filenode like 3276838.
- If the relation (as shown by pg_relation_size('tablename')) is 73GB,
it should consist of 327683, 327683.1, 327683.2, ..., 327683.72.
Check what additional files there are.
- Extra files can be left behind after a crash. Were there any?
Yours,
Laurenz Albe
Re: Size of /pgdata
On Sat, 2026-06-13 at 12:18 +0530, Raj wrote:
> I took one big sample table which is 73GB, and using relfilenode(327683) I tried finding the
> underlying file size du -sh 327683* and able to find the total size the files occpuing and
> found 113gb.......so for one table itself it is 40Gb additional...
Investigate more:
- Make sure that you don't get files from another filenode like 3276838.
- If the relation (as shown by pg_relation_size('tablename')) is 73GB,
it should consist of 327683, 327683.1, 327683.2, ..., 327683.72.
Check what additional files there are.
- Extra files can be left behind after a crash. Were there any?
Yours,
Laurenz Albe
Re: Size of /pgdata
On Mon, 2026-06-15 at 06:48 +0530, Raj wrote:
> On Mon, 15 Jun 2026, 01:00 Laurenz Albe, wrote:
> > On Sat, 2026-06-13 at 12:18 +0530, Raj wrote:
> > > I took one big sample table which is 73GB, and using relfilenode(327683) I tried finding the
> > > underlying file size du -sh 327683* and able to find the total size the files occpuing and
> > > found 113gb.......so for one table itself it is 40Gb additional...
> >
> > Investigate more:
> >
> > - Make sure that you don't get files from another filenode like 3276838.
> >
> > - If the relation (as shown by pg_relation_size('tablename')) is 73GB,
> > it should consist of 327683, 327683.1, 327683.2, ..., 327683.72.
> > Check what additional files there are.
> >
> > - Extra files can be left behind after a crash. Were there any?
>
> No, I am using the command du -ch 327101*
That reply does not make any sense to me.
Yours,
Laurenz Albe
Re: Size of /pgdata
On Mon, 2026-06-15 at 06:48 +0530, Raj wrote:
> On Mon, 15 Jun 2026, 01:00 Laurenz Albe, <laurenz.albe@cybertec.at> wrote:
> > On Sat, 2026-06-13 at 12:18 +0530, Raj wrote:
> > > I took one big sample table which is 73GB, and using relfilenode(327683) I tried finding the
> > > underlying file size du -sh 327683* and able to find the total size the files occpuing and
> > > found 113gb.......so for one table itself it is 40Gb additional...
> >
> > Investigate more:
> >
> > - Make sure that you don't get files from another filenode like 3276838.
> >
> > - If the relation (as shown by pg_relation_size('tablename')) is 73GB,
> > it should consist of 327683, 327683.1, 327683.2, ..., 327683.72.
> > Check what additional files there are.
> >
> > - Extra files can be left behind after a crash. Were there any?
>
> No, I am using the command du -ch 327101*
That reply does not make any sense to me.
Yours,
Laurenz Albe
Re: Size of /pgdata
On Mon, 2026-06-15 at 14:28 +0530, Raj wrote: > The command du -ch 327101* picks all files 327202, 327101.1, 327101.2 ....... > and give each of them sizes and also total size of all files listed as u see in my earlier screenshot I guess you didn't read my answer. Yours, Laurenz Albe
Re: Size of /pgdata
On Mon, 2026-06-15 at 14:28 +0530, Raj wrote:
> The command du -ch 327101* picks all files 327202, 327101.1, 327101.2 .......
> and give each of them sizes and also total size of all files listed as u see in my earlier screenshot
I guess you didn't read my answer.
Yours,
Laurenz Albe
Re: Size of /pgdata
On Tue, 2026-06-16 at 15:30 +0530, Raj wrote: > Moreover in a 3 node PATRONI...other two nodes are didn't have this issue.... > only one node has this issue of disk space difference of around 200gb Well, figure out which files are different. Perhaps you had a crash on one of the nodes that left some orphaned files around. Yours, Laurenz Albe
Re: Size of /pgdata
On Tue, 2026-06-16 at 15:30 +0530, Raj wrote:
> Moreover in a 3 node PATRONI...other two nodes are didn't have this issue....
> only one node has this issue of disk space difference of around 200gb
Well, figure out which files are different.
Perhaps you had a crash on one of the nodes that left some orphaned files around.
Yours,
Laurenz Albe
Re: Size of /pgdata
On Wed, 2026-06-17 at 11:47 +0530, Raj wrote: > The file count is same between them..but the file size varies. > > For example, assume node1 has issue and node2 is clean. > > 1. On both nodes file count is 76 > 2. File size varies for some of them > > Each files are either 1GB or 2GB and some or them are 1.2gb or 1.5gb or between 1-2gb ....by looking at files.. > In the problematic node, some of the 1gb files in node2 shows 2GB in node1...and also file size differences. making the difference. That is highly suspicious. What do you get if you run SHOW segment_size; Unless you built PostgreSQL yourself after changing the segment size, it should be 1GB. But then it would be impossible for some segment to be bigger than 1GB. Are you using some weird file system that reports file sizes wrongly? What does "ls -l" on one of these big files show? Yours, Laurenz Albe
AW: Size of /pgdata
In case you are using XFS, this might be related to the "Speculative preallocation" feature. We saw the same behavior on Redhat 9, with PostgreSQL 18 and XFS. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guide/migrating-ext4-xfs Speculative preallocation XFS uses speculative preallocation to allocate blocks past EOF as files are written. This avoids file fragmentation due to concurrent streaming write workloads on NFS servers. By default, this preallocation increases with the size of the file and will be apparent in "du" output. If a file with speculative preallocation is not dirtied for five minutes the preallocation will be discarded. If the inode is cycled out of cache before that time, then the preallocation will be discarded when the inode is reclaimed. If premature ENOSPC problems are seen due to speculative preallocation, a fixed preallocation amount may be specified with the -o allocsize=amount mount option. See also "man xfs" allocsize=size Sets the buffered I/O end-of-file preallocation size when doing delayed allocation writeout. Valid values for this option are page size (typically 4KiB) through to 1GiB, inclusive, in power-of-2 increments. The default behavior is for dynamic end-of-file preallocation size, which uses a set of heuristics to optimise the preallocation size based on the current allocation patterns within the file and the access patterns to the file. Specifying a fixed allocsize value turns off the dynamic behavior. Cheers William -----Ursprüngliche Nachricht----- Von: Laurenz Albe Gesendet: Mittwoch, 17. Juni 2026 13:34 An: Raj Cc: Pgsql-admin Betreff: Re: Size of /pgdata ACHTUNG: Diese Nachricht kommt von extern. Seien Sie kritisch beim Öffnen von Links und Anhängen. On Wed, 2026-06-17 at 11:47 +0530, Raj wrote: > The file count is same between them..but the file size varies. > > For example, assume node1 has issue and node2 is clean. > > 1. On both nodes file count is 76 > 2. File size varies for some of them > > Each files are either 1GB or 2GB and some or them are 1.2gb or 1.5gb or between 1-2gb ....by looking at files.. > In the problematic node, some of the 1gb files in node2 shows 2GB in node1...and also file size differences. making the difference. That is highly suspicious. What do you get if you run SHOW segment_size; Unless you built PostgreSQL yourself after changing the segment size, it should be 1GB. But then it would be impossible for some segment to be bigger than 1GB. Are you using some weird file system that reports file sizes wrongly? What does "ls -l" on one of these big files show? Yours, Laurenz Albe ________________________________ Disclaimer: Diese Nachricht und ihr eventuell angehängte Dateien sind nur für den Adressaten bestimmt. Sie kann vertrauliche oder gesetzlich geschützte Daten oder Informationen beinhalten. Falls Sie diese Nachricht irrtümlich erreicht hat, bitten wir Sie höflich, diese unter Ausschluss jeglicher Reproduktion zu löschen und die absendende Person zu benachrichtigen. Danke für Ihre Hilfe. This message and any attached files are for the sole use of the recipient named above. It may contain confidential or legally protected data or information. If you have received this message in error, please delete it without making any copies whatsoever and notify the sender. Thank you for your assistance.
Re: Size of /pgdata
In case you are using XFS, this might be related to the "Speculative preallocation" feature.
We saw the same behavior on Redhat 9, with PostgreSQL 18 and XFS.
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guide/migrating-ext4-xfs
Speculative preallocation
XFS uses speculative preallocation to allocate blocks past EOF as files are written. This avoids file fragmentation due to concurrent streaming write workloads on NFS servers. By default, this preallocation increases with the size of the file and will be apparent in "du" output. If a file with speculative preallocation is not dirtied for five minutes the preallocation will be discarded. If the inode is cycled out of cache before that time, then the preallocation will be discarded when the inode is reclaimed.
If premature ENOSPC problems are seen due to speculative preallocation, a fixed preallocation amount may be specified with the -o allocsize=amount mount option.
See also "man xfs"
allocsize=size
Sets the buffered I/O end-of-file preallocation size when doing delayed allocation writeout. Valid values for this option are page size (typically 4KiB) through to
1GiB, inclusive, in power-of-2 increments.
The default behavior is for dynamic end-of-file preallocation size, which uses a set of heuristics to optimise the preallocation size based on the current allocation
patterns within the file and the access patterns to the file. Specifying a fixed allocsize value turns off the dynamic behavior.
Cheers
William
-----Ursprüngliche Nachricht-----
Von: Laurenz Albe <laurenz.albe@cybertec.at>
Gesendet: Mittwoch, 17. Juni 2026 13:34
An: Raj <rajeshkumar.dba09@gmail.com>
Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Betreff: Re: Size of /pgdata
ACHTUNG: Diese Nachricht kommt von extern. Seien Sie kritisch beim Öffnen von Links und Anhängen.
On Wed, 2026-06-17 at 11:47 +0530, Raj wrote:
> The file count is same between them..but the file size varies.
>
> For example, assume node1 has issue and node2 is clean.
>
> 1. On both nodes file count is 76
> 2. File size varies for some of them
>
> Each files are either 1GB or 2GB and some or them are 1.2gb or 1.5gb or between 1-2gb ....by looking at files..
> In the problematic node, some of the 1gb files in node2 shows 2GB in node1...and also file size differences. making the difference.
That is highly suspicious.
What do you get if you run
SHOW segment_size;
Unless you built PostgreSQL yourself after changing the segment size, it should be 1GB. But then it would be impossible for some segment to be bigger than 1GB.
Are you using some weird file system that reports file sizes wrongly?
What does "ls -l" on one of these big files show?
Yours,
Laurenz Albe
________________________________
Disclaimer:
Diese Nachricht und ihr eventuell angehängte Dateien sind nur für den Adressaten bestimmt. Sie kann vertrauliche oder gesetzlich geschützte Daten oder Informationen beinhalten. Falls Sie diese Nachricht irrtümlich erreicht hat, bitten wir Sie höflich, diese unter Ausschluss jeglicher Reproduktion zu löschen und die absendende Person zu benachrichtigen. Danke für Ihre Hilfe.
This message and any attached files are for the sole use of the recipient named above. It may contain confidential or legally protected data or information. If you have received this message in error, please delete it without making any copies whatsoever and notify the sender. Thank you for your assistance.
AW: Size of /pgdata
By setting XFS allocsize, you switch off the dynamic behavior, and it becomes more predictable.
Another fix might be a switch to a different filesystem. E.g. ext4
While this XFS feature is a quite old one, I have seen it the first time kicking in quite heavily only on Red Hat 9, with PostgreSQL 18 and
only on the standby. Some times double the size for some files.
Von: Raj <rajeshkumar.dba09@gmail.com>
Gesendet: Mittwoch, 17. Juni 2026 20:46
An: Sescu William (SW0) <william.sescu@suva.ch>
Cc: Laurenz Albe <laurenz.albe@cybertec.at>; Pgsql-admin <pgsql-admin@lists.postgresql.org>
Betreff: Re: Size of /pgdata
Yes, this is what I have been asking. I also mentioned/asked earlier whether it's Abt XFS file system.
I checked bitmaps and under flags I could find 'unwritten preallocated extent'. --apparent-size says shows correct size.
Is the fix - setting up allocsize?
On Wed, 17 Jun 2026, 19:29 William Sescu (Suva), <william.sescu@suva.ch> wrote:
In case you are using XFS, this might be related to the "Speculative preallocation" feature.
We saw the same behavior on Redhat 9, with PostgreSQL 18 and XFS.
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/storage_administration_guide/migrating-ext4-xfs
Speculative preallocation
XFS uses speculative preallocation to allocate blocks past EOF as files are written. This avoids file fragmentation due to concurrent streaming write workloads on NFS servers. By default, this preallocation increases with the size of the file and will be apparent in "du" output. If a file with speculative preallocation is not dirtied for five minutes the preallocation will be discarded. If the inode is cycled out of cache before that time, then the preallocation will be discarded when the inode is reclaimed.
If premature ENOSPC problems are seen due to speculative preallocation, a fixed preallocation amount may be specified with the -o allocsize=amount mount option.
See also "man xfs"
allocsize=size
Sets the buffered I/O end-of-file preallocation size when doing delayed allocation writeout. Valid values for this option are page size (typically 4KiB) through to
1GiB, inclusive, in power-of-2 increments.
The default behavior is for dynamic end-of-file preallocation size, which uses a set of heuristics to optimise the preallocation size based on the current allocation
patterns within the file and the access patterns to the file. Specifying a fixed allocsize value turns off the dynamic behavior.
Cheers
William
-----Ursprüngliche Nachricht-----
Von: Laurenz Albe <laurenz.albe@cybertec.at>
Gesendet: Mittwoch, 17. Juni 2026 13:34
An: Raj <rajeshkumar.dba09@gmail.com>
Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Betreff: Re: Size of /pgdata
ACHTUNG: Diese Nachricht kommt von extern. Seien Sie kritisch beim Öffnen von Links und Anhängen.
On Wed, 2026-06-17 at 11:47 +0530, Raj wrote:
> The file count is same between them..but the file size varies.
>
> For example, assume node1 has issue and node2 is clean.
>
> 1. On both nodes file count is 76
> 2. File size varies for some of them
>
> Each files are either 1GB or 2GB and some or them are 1.2gb or 1.5gb or between 1-2gb ....by looking at files..
> In the problematic node, some of the 1gb files in node2 shows 2GB in node1...and also file size differences. making the difference.
That is highly suspicious.
What do you get if you run
SHOW segment_size;
Unless you built PostgreSQL yourself after changing the segment size, it should be 1GB. But then it would be impossible for some segment to be bigger than 1GB.
Are you using some weird file system that reports file sizes wrongly?
What does "ls -l" on one of these big files show?
Yours,
Laurenz Albe
________________________________
Disclaimer:
Diese Nachricht und ihr eventuell angehängte Dateien sind nur für den Adressaten bestimmt. Sie kann vertrauliche oder gesetzlich geschützte Daten oder Informationen beinhalten. Falls Sie diese Nachricht irrtümlich erreicht hat, bitten wir Sie höflich, diese unter Ausschluss jeglicher Reproduktion zu löschen und die absendende Person zu benachrichtigen. Danke für Ihre Hilfe.
This message and any attached files are for the sole use of the recipient named above. It may contain confidential or legally protected data or information. If you have received this message in error, please delete it without making any copies whatsoever and notify the sender. Thank you for your assistance.