Обсуждение: reduce size of logs stored by buildfarm
Hi, I just noticed that one regress log file for the pg_dump test is about 7MB long because it contains totally unnecessary dump files in the regress log output. This is because the tests run pg_dump of schema pg_catalog without redirecting the output anywhere, which means it goes to stdout, and then `prove` helpfully puts it in the regress log file. This is completely pointless. We can make the log file 1 MB instead by adding a --file parameter to that command, as 0001 attached. We also have a couple of tests that leave rather large server logs behind that are perhaps not very useful. I would propose to run those servers with log_statement=off and avoid some of the bloat. 0002 does that. Another possible approach, instead of individually tweaking t/*.pl files to add 'log_statements=none', would be to set it that way in PostgreSQL::Test::Cluster globally. Right now we have log_statements=all there, but maybe that was not a great idea. I also noticed that an oauth test file contains a couple MBs of a gigantic string of just 'x'. I suppose that will compress well (with gzip at least, the 2 MB file becomes 8 kB). Still, it's kinda ridiculous and useless to bloat a 67 kB file to 2 MB that way. The problem here is that we print 'sending JSON response' and the verbatim response, included the 1 MB of "x" as '_pad_'. Can we supress this? 0003 does that by simply cutting the string short at 10k, which reduces the size of the log from 2 MB to some 86 kB. Maybe there are better ways to deal with this? Jacob? With these three patches, we go from having 62 MB bytes in files with "log" in their names under testrun/ (except those that have "catalog") to 30 MB. I think that's not a bad reduction. -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/ Are you not unsure you want to delete Firefox? [Not unsure] [Not not unsure] [Cancel] http://smylers.hates-software.com/2008/01/03/566e45b2.html
Вложения
Hi, On 2025-11-25 14:30:33 +0100, Álvaro Herrera wrote: > I just noticed that one regress log file for the pg_dump test is about > 7MB long because it contains totally unnecessary dump files in the > regress log output. This is because the tests run pg_dump of schema > pg_catalog without redirecting the output anywhere, which means it goes > to stdout, and then `prove` helpfully puts it in the regress log file. > This is completely pointless. We can make the log file 1 MB instead by > adding a --file parameter to that command, as 0001 attached. Yea, that makes no sense. > We also have a couple of tests that leave rather large server logs > behind that are perhaps not very useful. I would propose to run those > servers with log_statement=off and avoid some of the bloat. 0002 does > that. > > Another possible approach, instead of individually tweaking t/*.pl files > to add 'log_statements=none', would be to set it that way in > PostgreSQL::Test::Cluster globally. Right now we have > log_statements=all there, but maybe that was not a great idea. I think that be a bad idea - many tests are essentially undebuggable that way because there's no other way to correlate the regress_log and the server log than the statement log. TBH, I have even needed the pg_dump log_statement output on the buildfarm to debug issues :( I am against doing this for 027_stream_regress.pl, it's definitely undebuggable without the statement logs. Greetings, Andres Freund
=?utf-8?Q?=C3=81lvaro?= Herrera <alvherre@kurilemu.de> writes:
> With these three patches, we go from having 62 MB bytes in files with
> "log" in their names under testrun/ (except those that have "catalog")
> to 30 MB. I think that's not a bad reduction.
I wonder how much this overlaps with what Andrew just did
to the BF client [1].
regards, tom lane
[1] https://www.postgresql.org/message-id/601bb91d-b55d-4fa6-bb57-c2126fb22620%40dunslane.net
On 2025-11-25 Tu 10:37 AM, Tom Lane wrote: > =?utf-8?Q?=C3=81lvaro?= Herrera <alvherre@kurilemu.de> writes: >> With these three patches, we go from having 62 MB bytes in files with >> "log" in their names under testrun/ (except those that have "catalog") >> to 30 MB. I think that's not a bad reduction. > I wonder how much this overlaps with what Andrew just did > to the BF client [1]. > > > [1] https://www.postgresql.org/message-id/601bb91d-b55d-4fa6-bb57-c2126fb22620%40dunslane.net > > Pretty much none of it, although both Alvaro and I are clearly motivated by the same thing, namely disk space pressure on the buildfarm server. I think his patches 1 and 3 seem like no-brainers. I did two things in the BF client: . prevent uploading redundant copies of the same file in meson builds (this was a bug) . inhibit loading postmaster log files from installcheck runs and cross-version upgrade runs, but only if there hasn't been a failure. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
On 2025-Nov-25, Andrew Dunstan wrote: > I think his patches 1 and 3 seem like no-brainers. On 2025-Nov-25, Andres Freund wrote: > On 2025-11-25 14:30:33 +0100, Álvaro Herrera wrote: > > I just noticed that one regress log file for the pg_dump test is about > > 7MB long because it contains totally unnecessary dump files in the > > regress log output. This is because the tests run pg_dump of schema > > pg_catalog without redirecting the output anywhere, which means it goes > > to stdout, and then `prove` helpfully puts it in the regress log file. > > This is completely pointless. We can make the log file 1 MB instead by > > adding a --file parameter to that command, as 0001 attached. > > Yea, that makes no sense. Thanks, pushed 0001. -- Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/ "Postgres is bloatware by design: it was built to house PhD theses." (Joey Hellerstein, SIGMOD annual conference 2002)
On Tue, Nov 25, 2025 at 5:30 AM Álvaro Herrera <alvherre@kurilemu.de> wrote: > I also noticed that an oauth test file contains a couple MBs of a > gigantic string of just 'x'. I suppose that will compress well (with > gzip at least, the 2 MB file becomes 8 kB). Still, it's kinda > ridiculous and useless to bloat a 67 kB file to 2 MB that way. Sorry about that. > 0003 does that by simply cutting the string short at 10k, which reduces > the size of the log from 2 MB to some 86 kB. Maybe there are better > ways to deal with this? Jacob? Only thing I don't like about this is that the JSON you need for debugging might be after the useless padding. Attached is a patch that does things more surgically, tested against Python 3.6, and I'm running it through the CI now. Thanks, --Jacob
Вложения
On 2025-Nov-25, Jacob Champion wrote: > On Tue, Nov 25, 2025 at 5:30 AM Álvaro Herrera <alvherre@kurilemu.de> wrote: > > I also noticed that an oauth test file contains a couple MBs of a > > gigantic string of just 'x'. I suppose that will compress well (with > > gzip at least, the 2 MB file becomes 8 kB). Still, it's kinda > > ridiculous and useless to bloat a 67 kB file to 2 MB that way. > > Sorry about that. No need! > > 0003 does that by simply cutting the string short at 10k, which reduces > > the size of the log from 2 MB to some 86 kB. Maybe there are better > > ways to deal with this? Jacob? > > Only thing I don't like about this is that the JSON you need for > debugging might be after the useless padding. Attached is a patch that > does things more surgically, tested against Python 3.6, and I'm > running it through the CI now. Brilliant, thanks. -- Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/ "You're _really_ hosed if the person doing the hiring doesn't understand relational systems: you end up with a whole raft of programmers, none of whom has had a Date with the clue stick." (Andrew Sullivan) https://postgr.es/m/20050809113420.GD2768@phlogiston.dyndns.org
On Tue, Nov 25, 2025 at 10:46 AM Álvaro Herrera <alvherre@kurilemu.de> wrote: > > Only thing I don't like about this is that the JSON you need for > > debugging might be after the useless padding. Attached is a patch that > > does things more surgically, tested against Python 3.6, and I'm > > running it through the CI now. > > Brilliant, thanks. (This was pushed last week; forgot to follow up here.) Thanks, --Jacob