Greetings,
* David Rowley (dgrowleyml@gmail.com) wrote:
> On Thu, 11 Jun 2020 at 10:02, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Thomas Munro <thomas.munro@gmail.com> writes:
> > > I've been doing that in a little database that pulls down the results
> > > and analyses them with primitive regexes. First I wanted to know the
> > > pass/fail history for each individual regression, isolation and TAP
> > > script, then I wanted to build something that could identify tests
> > > that are 'flapping', and work out when the started and stopped
> > > flapping etc. I soon realised it was all too noisy, but then I
> > > figured that I could fix that by detecting crashes. So I classify
> > > every top level build farm run as SUCCESS, FAILURE or CRASH. If the
> > > top level run was CRASH, than I can disregard the individual per
> > > script results, because they're all BS.
> >
> > If you can pin the crash on a particular test script, it'd be useful
> > to track that as a kind of failure. In general, though, both crashes
> > and non-crash failures tend to cause collateral damage to later test
> > scripts --- if you can't filter that out then the later scripts will
> > have high false-positive rates.
>
> I guess the fact that you've both needed to do analysis on individual
> tests shows that there might be a call for this beyond just recording
> the test's runtime.
>
> If we had a table that stored the individual test details, pass/fail
> and just stored the timing information along with that, then, even if
> the timing was unstable, it could still be useful for some analysis.
> I'd be happy enough even if that was only available as a csv file
> download. I imagine the buildfarm does not need to provide us with
> any tools for doing analysis on this. Ideally, there would be some
> run_id that we could link it back to the test run which would give us
> the commit SHA, and the animal that it ran on. Joining to details
> about the animal could be useful too, e.g perhaps a certain test
> always fails on 32-bit machines.
>
> I suppose that maybe we could modify pg_regress to add a command line
> option to have it write out a machine-readable file, e.g:
> testname,result,runtime\n, then just have the buildfarm client ship
> that off to the buildfarm server to record in the database.
That seems like it'd be the best approach to me, though I'd defer to
Andrew on it.
By the way, if you'd like access to the buildfarm archive server where
all this stuff is stored, that can certainly be arranged, just let me
know.
Thanks,
Stephen