On Tue, Apr 15, 2014 at 2:47 PM, Josh Berkus <josh@agliodbs.com> wrote:
Hackers,
I think 9.3 has given us evidence that our users aren't giving new versions of PostgreSQL substantial beta testing, or if they are, they aren't sharing the results with us.
A lot of the bugs that turned up are not the kind I would expect to have been found in most beta testing done by non-hacking users. Weren't they mostly around rare race conditions, crash recovery, and freezing?
How can we make beta testing better and more effective? How can we get more users to actually throw serious workloads at new versions and share the results?
"pgsql-hackers: bugs, questions, and successful test reports are welcome here if you are already subscribed to pgsql-hackers. Note that pgsql-hackers is a high-traffic mailing list with a lot of development discussion."
So successful reports are welcome, provided that you are willing to subscribe to a list that generates tons of noise you won't understand. That doesn't sound all that welcoming. (I already am subscribed, but I still usually don't report successful tests, because "yeah, I did a bunch of stuff, and nothing failed in an obvious way" just doesn't sound very useful, and it is hard to get motivated to write up an exhaustive description of a test that doesn't prove anything anyway--maybe if I did for a few more hours, it would have found a problem.)
If we want to know how much beta testing is really going on, perhaps we could do a survey asking people whether they did any beta testing, and if so whether they reported the results. Otherwise it would be hard to distinguish "We aren't doing enough testing" from "We do lots of testing, but it isn't strenuous enough to find the problems, or is testing the wrong aspects of the system".