Fabien COELHO <coelho@cri.ensmp.fr> writes:
> So this means somehow giving up on coverage, because of one host which can
> fail under very unusual testing conditions once in a while.
You are attacking a straw man. There is no reason whatever to believe
that the problem is confined to one host. We have quite a number of slow
buildfarm hosts: some are using CLOBBER_CACHE_ALWAYS, some are overloaded,
some are just plain slow. Also, regression test failures outside the
context of our buildfarm are just as bad if not worse. I still remember
the pain I had with mysql's regression tests, which routinely fell over
in Red Hat's package buildfarm even though they always succeeded on a
personal workstation with nothing else going on. I do not want our own
packagers to have that kind of experience with Postgres.
> I would prefer to keep the test and have a warning instead, something
> like "ok, although a test which is allowed to rarely fail failed", but at
> least the feature are tested most of the time.
If the failures are ignored, how much of a test have you really got?
Nobody inquires into nominally-passed buildfarm runs to see what
actually happened.
In the TAP-test context, you do have quite a bit of freedom to decide
what is a pass and what is a fail. So maybe the path forward is to be
more flexible about the pass conditions for this test. But I can't
help thinking that these results have exposed some outright bugs too.
regards, tom lane
--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers