Reducing buildfarm disk usage: remove temp installs when done

Поиск
Список
Период
Сортировка
От Tom Lane
Тема Reducing buildfarm disk usage: remove temp installs when done
Дата
Msg-id 18617.1421621291@sss.pgh.pa.us
обсуждение исходный текст
Ответы Re: Reducing buildfarm disk usage: remove temp installs when done  (Michael Paquier <michael.paquier@gmail.com>)
Re: Reducing buildfarm disk usage: remove temp installs when done  (Andrew Dunstan <andrew@dunslane.net>)
Re: Reducing buildfarm disk usage: remove temp installs when done  (Andres Freund <andres@2ndquadrant.com>)
Список pgsql-hackers
One of the biggest causes of buildfarm run failures is "out of disk
space".  That's not just because people are running buildfarm critters
on small slow machines; it's because "make check-world" is an enormous
space hog.  Some numbers from current HEAD:

clean source tree:        120MB
built source tree:        400MB
tree after make check-world:    3GB

(This is excluding ~250MB for one's git repo.)

The reason for all the bloat is the temporary install trees that we
create, which tend to eat up about 100MB apiece, and there are dozens
of them (eg, one per testable contrib module).  Those don't get removed
until the end of the test run, so the usage is cumulative.

The attached proposed patch removes each temp install tree as soon as
we're done with it, in the normal case where no error was detected.
This brings the peak space usage down from ~3GB to ~750MB.

To make things better in the buildfarm, we'd have to back-patch this into
all active branches, but I don't see any big problem with doing so.

Any objections?

            regards, tom lane

diff --git a/src/test/regress/pg_regress.c b/src/test/regress/pg_regress.c
index caae3f0..ee3b80b 100644
*** a/src/test/regress/pg_regress.c
--- b/src/test/regress/pg_regress.c
*************** regression_main(int argc, char *argv[],
*** 2668,2673 ****
--- 2668,2686 ----
          stop_postmaster();
      }

+     /*
+      * If there were no errors, remove the temp installation immediately to
+      * conserve disk space.  (If there were errors, we leave the installation
+      * in place for possible manual investigation.)
+      */
+     if (temp_install && fail_count == 0 && fail_ignore_count == 0)
+     {
+         header(_("removing temporary installation"));
+         if (!rmtree(temp_install, true))
+             fprintf(stderr, _("\n%s: could not remove temp installation \"%s\": %s\n"),
+                     progname, temp_install, strerror(errno));
+     }
+
      fclose(logfile);

      /*

В списке pgsql-hackers по дате отправления:

Предыдущее
От: Tom Lane
Дата:
Сообщение: Re: Re: Better way of dealing with pgstat wait timeout during buildfarm runs?
Следующее
От: Michael Paquier
Дата:
Сообщение: Re: Reducing buildfarm disk usage: remove temp installs when done