On 1/22/16 12:14 PM, Andres Freund wrote:
> On 2016-01-22 08:40:28 -0600, Jim Nasby wrote:
>> Ideally reviewers shouldn't be doing any testing, because the tests
>> that are part of the patch should answer every question they would
>> have, but I don't see that happening until we have a separate
>> automation-only target that we don't care how long it takes to run.
>
> I think that's completely wrong.
>
> Yes, more tests are good, and we need a place for longer running
> tests. But assuming that every patch author will create a testsuite that
> covers every angle is just about akin to assuming every submitter will
> deliver perfect, bug free code. And we know how well that turns out.
>
> I think actively trying to break a feature, and postgres in general, is
> one of the most important tasks of reviewers and testers. And with that
> I don't mean trying to run "make check". Look e.g. at the tests Jeff
> Janes has performed, what the recent plug tests of Tomas Vondra brought
> to light, or at what the full page write checker tool of Heikki's
> showed.
IIRC Jeff's tests are scripted and obviously the page write checker is
as well. I don't recall the exact methodology Tomas was using but I
suspect it could also be scripted if you had a way to pull the plug via
software (via a power management unit or maybe kill -9 of a VM). All of
that is stuff that can and should be automated. Presumably it won't ever
be part of the Makefile tests, but that's fine. Heck, the test scripts
could stay in completely separate repos.
Where the code lives isn't the issue; it's getting stuff like this
automated so humans can go back do doing things that can't be automated.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com