On Fri, Apr 21, 2017 at 10:34:58PM -0700, Andres Freund wrote:
> I've a bunch of tests, but I don't quite know whether we can expose all
> of them via classical tests. There are several easy ones that I
> definitely want to add (import "empty" snapshot; import snapshot with
> running xacts; create snapshot, perform some ddl, import snapshot,
> perform some ddl, check things work reasonably crazy), but there's
> enough others that are just probabilistic. I was wondering about adding
> a loop that simply runs for like 30s and then quits or such, but who
> knows.
If the probabilistic test catches the bug even 5% of the time in typical
configurations, the buildfarm will rapidly identify any regression. I'd
choose a 7s test that detects the bug 5% of the time over a 30s test that
detects it 99% of the time. (When I wrote src/bin/pgbench/t/001_pgbench.pl
for a probabilistic bug, I sized that test to finish in 1s and catch its bug
half the time. In its case, only two buildfarm members were able to
demonstrate the original bug, so 5% detection would have been too low.)