Re: Two weeks to feature freeze

On Thu, 26 Jun 2003, Bruce Momjian wrote:

> 
> See my recent commit of src/tools/pgtest.  It might be a good start.

Yes this is a good start. This is a little concerning though:

pg_ctl stop
rm -rf "$PGDATA"

Perhaps a warning is warranted (ie, $PGDATA shouldn't point to your
production data cluster)?

On another point, I have some ideas for Kevin and others interested in
automated testing. Dann, Tom and others have voiced concern about the
nature of testing itself: progammers testing for bugs they've solved; the
difficulty/impossibility of testing for conditions you are unaware of,
etc.

ISTM that most of the bugs which aren't caught by the programmer, peer
review and regresion test are revealed by users throwing data into a new
version or a version different to that they are running in production and
then running their existing code against it. That is, bugs are revealed by
usage which developers did not foresee or did not think to test.

So, if we had an automated testing framework which was extensible,
postgres users could create testing scripts which simultate their
application, with their application data (real or created specifically
for the test). The win for users is that they can have their data/SQL
tested on a variety of platforms, on new versions of postgres and the win
for developers/testers is exposure of the code to unexpected usage.

There will need to be checks and balances involved (select 1; is a pretty
ordinary test), size limits/configurable thresholds for run times, and a
repository of test results.

Naturally, managing this could be quite time consuming if data formats
change etc. but if people are keen...

Gavin

> 
> ---------------------------------------------------------------------------
> 
> Gavin Sherry wrote:
> > On Thu, 26 Jun 2003, Kevin Brown wrote:
> > 
> > > The Hermit Hacker wrote:
> > > > On Wed, 25 Jun 2003, Kevin Brown wrote:
> > > > 
> > > > > So...would it make sense to create a gborg project to which people who
> > > > > have written their own test suites can contribute whatever code and data
> > > > > they feel comfortable releasing?  As a gborg project, it would be
> > > > > separate from the main PG distribution and would thus have no impact on
> > > > > the build process or anything like that.  But at the same time, if there
> > > > > are any ideas on testing that people have had, they could be shared with
> > > > > others through that mechanism.
> > > > >
> > > > > And any tests which prove to be particularly useful could make their way
> > > > > into the PG distribution if people here wish.
> > > > >
> > > > > Of course, like anything else this could be a bad (or perhaps redundant)
> > > > > idea.  :-)
> > > > 
> > > > It doesn't sound like a bad idea ... but, it pretty much comes down to the
> > > > original thread: are you willing to step up and maintain such a project?
> > > 
> > > Yes, I am ("how hard can it be?", he asks himself, knowing all the
> > > while that it's a really bad idea to be asking that question.  :-).
> > > But I haven't the faintest idea of how or where to even start, so
> > > pointers would be appreciated.
> > 
> > Create/modify a script to automate some kind of download/sync, test and
> > send failure results somewhere. Make it extensible, so that other tests
> > can be easily added -- preferable in a self contained way. It should grow
> > from there.
> > 
> > Gavin
> > 
> > 
> > ---------------------------(end of broadcast)---------------------------
> > TIP 5: Have you checked our extensive FAQ?
> > 
> >                http://www.postgresql.org/docs/faqs/FAQ.html
> > 
> 
>