Re: Add more regression tests for dbcommands

On Wed, Jun 26, 2013 at 11:33 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Josh Berkus <josh@agliodbs.com> writes:
>> On 06/26/2013 12:08 PM, Fabien COELHO wrote:
>>> I have been suggesting something upon that line in some of the reviews
>>> I've posted about Robins non regression tests, if they were to be
>>> rejected on the basis that they add a few seconds for checks. They are
>>> well made to test corner cases quite systematically, and I feel that it
>>> would be sad if they were lost.
>
>> My thinking was that someone should add all of his new tests at once,
>> and then see how much of a time difference they make.  If it's 7
>> seconds, who cares?
>
> Making that measurement on the current set of tests doesn't seem to me
> to prove much.  I assume Robins' eventual goal is to make a significant
> improvement in the tests' code coverage across the entire backend, and
> what we see submitted now is just as much as he's been able to do yet
> in that line.  So even if the current cost is negligible, I don't think
> it'll stay that way.

Well, this is a discussion we've had before.  I'm certainly in favor
of having a bigger set of regression tests, so that people can just
run the small suite if they want to.  But I'm not keen on pushing
tests into that extended suite when they run in a fractions of a
second.  That's likely to create more hassle than it solves, since
people will forget to update the expected outputs if they don't run
those tests, and if they do run those tests, then having them in a
separate suite isn't saving anything.  The place for a separate test
suite, IMHO, is when you have tests that run for a long time
individually.

FWIW, internally to EDB, we have it broken down just about exactly
that way.  Our core set of regression tests for PPAS takes about 5
minutes to run on my laptop, and unfortunately quite a bit longer on
some older laptops.  It's a nuisance, but it's a manageable nuisance,
and it finds a lot of coding errors.  Then we have separate schedules
for test suites that contain long-running tests or have dependencies
on which compiler flags you use; most developers don't run that suite
regularly, but it gets run every night.  That keeps those annoyances
out of the foreground path of developers.  I really don't see why we
shouldn't take the same approach here.

I don't think we want as large a regression test suite for PostgreSQL
as we have for PPAS, because among other things, PostgreSQL doesn't
have a paid staff of people who can be roped into helping manage it
when needed.  But the management effort for the kind of expansion that
Robins Tharakan is proposing here is not going to be all that large,
at least if my EDB experience is any indication.  And I think it will
help find bugs.  I also think (and this is also something we've seen
internally) that part of the value of a regression suite is that it
exposes when you've changed the behavior.  The regression tests fail
and you have to change the expected outputs; fine.  But now it's much
less likely that you're going to make those kinds of changes without
*realizing* that you've changed the behavior.  I don't have a specific
example top of mind right now, but I am pretty sure there have been at
least a few cases where PostgreSQL changes have had knock-on
consequences that weren't obvious at the time they were made.  The
kind of work that Robins is doing here is not going to fix that
problem entirely, but I think it will help, and I think that has a lot
of value.

So I'd like to endorse Josh's idea: subject to appropriate review,
let's add these test cases.  Then, if it really turns out to be too
burdensome, we can take them out, or figure out a sensible way to
split the suite.  Pushing all of Robins work into a secondary suite,
or throwing it out altogether, both seem to me to be things which will
not be to the long-term benefit of the project.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company