On Wednesday 10 December 2008 19:36:38 Tom Lane wrote:
> Zdenek Kotala <Zdenek.Kotala@Sun.COM> writes:
> > Tom Lane napsal(a):
> >> No, the standard way to deal with such issues is to set up two buildfarm
> >> members.
> >
> > I think current infrastructures is not good for it. For example I would
> > like to compile postgres on one machine with three different compiler and
> > in 32 or 64 mode. Should I have 6 animals?
>
> Yes.
I have to say, I have concerns similar to Zdenek's. Setting up a load of
different animals for every altered configuration makes it difficult to tell
which configurations are actually related.
I have been thinking about test coverage recently and analyzed bugs and so on.
To get more confidence beyond a random (not even truly random) subset of
platforms and options we should really be building with a lot more
combinations of
- compilers
- compiler options
- configure options
- run time options
(- more tests of other code areas, but that is a different problem)
Note, for example, that downstream binary packages are almost never built with
default or near-default compiler options, and of course production
installations are hopefully never run with the default run-time
configuration. Essentially, we are not really testing what the users are
running.
To cover reality better, I can easily imagine that a single platform (say,
CPU, OS, bitness, and compiler) should do at least fifty different test runs
in different combinations. There, we'd also have resource problems, but some
people have machines that can do that (and want to do that). How can we
accomodate that today?
A coincidental trouble with this is that I find the animal names to be
increasingly difficult to process and remember. They are basically just line
noise to me at this point. Other non-biologists might feel the same. And we
might eventually run out of reasonable names.
> That simply complicates everything --- the reporting infrastructure,
> identifying which case failed, etc --- without actually improving
> anything.
I don't think it has to be that complicated. We could probably augment the
naming scheme like "animal/foo" or "animal/12" or something like that.