On Wed, Aug 26, 2009 at 5:12 PM, Peter Eisentraut<peter_e@gmx.net> wrote:
> On ons, 2009-08-26 at 14:26 -0400, Robert Haas wrote:
>> Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months
>> doesn't necessarily buy you much, either. I'm good at focused
>> activity - but there was nothing focused about 8.4 beta that I could
>> see. Maybe we need some kind of TestFest process.
>
> Yeah, exactly. I can't imagine end users would know what to do during
> beta. Even assuming that you have release notes at the beginning of
> beta, you can't expect people to go through every item and do a formal
> test for it. Surely it's been tested before, else it would not be in
> the release, right?
I would sure hope so. Testing features individually makes a whole lot
more sense to me than testing the release as a whole. Just trying a
bunch of random stuff and seeing if anything breaks is not a very
productive activity. On the other hand, testing individual features
is frequently very productive, but it's my understanding of the way PG
does things that that is supposed to happen before the patch is
committed.
It appears to me that most of the really nasty bugs that have been
found in 8.4.0 relate to one of the following three things, each of
which seems to be related to multiple back-branch commits.
1. SEMI/ANTI join support.
2. running the bgwriter during recovery (infrastructure changes for recovery)
3. deadman switch
Maybe some of these weren't tested well enough prior to commit? Or
perhaps they're just more significant changes and therefore likely
spots for rough edges.
I think there is a lot of merit (as Andrew suggests) in running a
production application on a beta version of the database just to see
if anything funny happens. But insisting that all PG developers stop
doing development to focus ONLY on that activity doesn't seem very
reasonable: not everyone is well-placed to do that kind of experiment,
or cares to do so. Conversely, there are many people who are NOT
developers who ARE well-placed to beta test (for example, Kevin
Grittner does a lot more testing than he does development, and I think
there are few people on this mailing list who would argue that the
quality of that testing is any less than kickass).
...Robert