Simon Riggs writes:
> It's good that the input is fuzzed, but there needs to be a way to re-run
> identical fuzzing or a way to backtrack to find what broke. Not much point
> finding bugs we can't identify later.
sqlsmith is deterministic and allows re-generating a sequence of random
queries with the --seed argument. Finding a testing methodology that
ensures a repeatable server-side is a harder problem though.
One would have to avoid touching any kind of concurrency, disable
autovacuum, autoanalyze and invoke explicit analyzes/vacuums in concert
with query generation. Further, one would have to avoid any kind of
concurrency while testing. Even then, 1% of the queries run into a
statement_timeout due to randomly generated excessive cross joins. If a
timeout just barely happens, it might not do so on the repeated run and
the deterministic state is gone from then on. I'm afraid this list is
not complete yet.
I didn't think the effort of creating this kind of clean-room testing
was worth it. If reports of failed assertions with backtrace without a
recipe to reproduce them are a nuisance, I'll avoid them in the future.
regards,
Andreas