Обсуждение: TESTING (was: RE: More vacuum.c refactoring )

> -----Original Message-----
> From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Tom Lane
> Sent: Thursday, June 10, 2004 2:19 PM
> To: Manfred Koizar
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] More vacuum.c refactoring
>
>
> Manfred Koizar <mkoi-pg@aon.at> writes:
> > This code is very similar to vacuum_page().  The major
> difference is
> > that vacuum_page() uses vacpage->offsets while the code in
> > repair_frag() looks for MOVED_OFF bits in tuple headers.
> AFAICS the
> > tuples with the MOVED_OFF bit set are exactly those referenced by
> > vacpage->offsets.
>
> This does not make me comfortable.  You *think* that two
> different bits of code are doing the same thing, so you want
> to hack up vacuum.c?  This module is delicate code --- we've
> had tons of bugs there in the past
> --- and no I have zero confidence that passing the regression
> tests proves anything, because all those prior bugs passed
> the regression tests.

Then why didn't those bugs get added to the regression?  That has been
standard procedure in every place that I have ever worked.
We have 7000+ tests in our CONNX regression suite that take one week to
run, on an array of dozens of computers from micros to mainframes.
Besides finding quickly if you have reintroduced a problem, it will also
ferret out lots of newly introduced problems.
It seems that MySQL has some sort of extensive test, from looking at
their site.  Maybe the Pgsql group could simply cannibalize it.

Perhaps your regression test is really a sanity check, rather than a
regression test. After all, the meaning of 'regression' itself demands
that you introduce new tests based upon old failures.

I seem to recall that someone was porting the NIST suite to PostgreSQL.
What ever happened to that effort?

"Dann Corbit" <DCorbit@connx.com> writes:
>> --- and no I have zero confidence that passing the regression 
>> tests proves anything, because all those prior bugs passed 
>> the regression tests.

> Then why didn't those bugs get added to the regression?

Because there wasn't any reasonable way to make them reproducible.

The set of things we can test in the regression tests is only a small
fraction of the interesting properties of Postgres.  This is
unfortunate but ranting about "standard practice" doesn't change it.

> I seem to recall that someone was porting the NIST suite to PostgreSQL.
> What ever happened to that effort?

It was done and we fixed a couple of bugs based on it (the one I can
think of offhand had to do with semantics of aggregate functions in
sub-selects).  I don't think there's anything more to be learned there.
        regards, tom lane

> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sent: Friday, June 11, 2004 2:35 PM
> To: Dann Corbit
> Cc: Manfred Koizar; pgsql-hackers@postgresql.org
> Subject: Re: TESTING (was: RE: [HACKERS] More vacuum.c refactoring )
>
>
> "Dann Corbit" <DCorbit@connx.com> writes:
> >> --- and no I have zero confidence that passing the regression
> >> tests proves anything, because all those prior bugs passed
> >> the regression tests.
>
> > Then why didn't those bugs get added to the regression?
>
> Because there wasn't any reasonable way to make them reproducible.
>
> The set of things we can test in the regression tests is only
> a small fraction of the interesting properties of Postgres.
> This is unfortunate but ranting about "standard practice"
> doesn't change it.
>
> > I seem to recall that someone was porting the NIST suite to
> > PostgreSQL. What ever happened to that effort?
>
> It was done and we fixed a couple of bugs based on it (the
> one I can think of offhand had to do with semantics of
> aggregate functions in sub-selects).  I don't think there's
> anything more to be learned there.

It is reassuring to know that it passed with flying colors.

Can I get the ported version?

I would love to play with it.

"Dann Corbit" <DCorbit@connx.com> writes:
>> It was done and we fixed a couple of bugs based on it (the 
>> one I can think of offhand had to do with semantics of 
>> aggregate functions in sub-selects).  I don't think there's 
>> anything more to be learned there.

> It is reassuring to know that it passed with flying colors.

"Passed with flying colors" might be overstating it --- but the other
things it found were stuff we already knew about, eg, unimplemented
features, identifier case folding, that kind of thing.
        regards, tom lane