On Sat, Oct 18, 2014 at 1:39 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
David Rowley <dgrowleyml@gmail.com> writes: > On Fri, Oct 17, 2014 at 3:34 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> I don't want to go there. It would be a lot better to expend the effort >> on a better regression testing infrastructure that wouldn't *need* >> bitwise-identical output across platforms. (mysql is ahead of us in that >> department: they have some hacks for selective matching of the output.)
> Perhaps we could introduce some sort of wildcard matching in the regression > tests. So that we could stick something like: > Execution time: * ms > Into the expected results, though, probably we'd need to come up with some > wildcard character which is a bit less common than *
I was imagining that we might as well allow regexp matching, so you could be as specific as
Execution time: \d+\.\d+ ms
if you had a mind to. But with or without that, it would be difficult to pick a meta-character that never appears in expected-output files today. What we'd probably want to do (again, I'm stealing ideas from what I remember of the mysql regression tests) is add metasyntax to switch between literal and wildcard/regexp matching. So perhaps an expected file could contain something like
-- !!match regexp ... expected output including regexps ... -- !!match literal ... normal expected output here
That seems better, that way we'd be safer from accidentally matching when we shouldn't
Not sure how we get there without writing our own diff engine though :-(.
I had imagined that we wouldn't need this, but perhaps my workflow is just different from yours. When I make changes which make tests fail for a valid reason I'd use beyondcompare to cherrypick the actual back into the expected, but I suppose others might just apply the diff into the expected.... Umm, but then wouldn't you just copy the whole actual file over to expected?... So why do we need diffs? Couldn't this matching tool just report where the first non-matching line appeared in the file? We could then manually diff the files and just ignore the !!match stuff and the regex differences.