* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Realistically though, how much would code coverage info have helped?
> Code coverage on a back branch would not have told you about whether
> it leaves blobs behind in the final regression DB state. Code coverage
> on HEAD might have helped you notice some aspects of this failure, but
> it would not have told you about the same query failing before 7.4
> that worked later.
The code coverage report is exactly what I was using to figure out what
was being tested in pg_dump and what wasn't. Many of the tests that are
included in the new TAP testing framework that I wrote for pg_dump were
specifically to provide code coverage and did improve the report.
If the regression tests in older versions were updated to make sure that
all the capabilities of pg_dump in those versions were tested then my
testing with the regression test databases would have shown that the
newer version of pg_dump wasn't handling those cases correctly. That
would require more comprehensive testing to be done in those
back-branches though, which would require more than just the code
coverage tool being included, that's true.
Another approach to this would be to figure out a way for the newer
testing framework in HEAD to be run against older versions, though we'd
need to have a field which indicates which version of PG a given test
should be run against as there are certainly tests of newer capabilities
than older versions supported.
Ultimately, I'm afraid we may have to just punt on the idea of this kind
of testing being done using the same testing structure that exists in
HEAD and is used in the buildfarm. That would be unfortunate, but I'm
not quite sure how you could have a buildfarm member than runs every
major version between 8.0 and HEAD and knows how to tell the HEAD
build-system what all the ports are for all those versions to connect to
and run tests against.
Thanks!
Stephen