Greg Stark writes:
> On Sat, Dec 12, 2015 at 8:30 PM, Andreas Seltenreich <seltenreich@gmx.de> wrote:
> When you hit the timeout is this implemented in your fuzzer or using
> statement_timeout? If the former, can you add a statement_timeout of
> just short of the timeout in the fuzzer and find cases where the
> planner might not be calling CHECK_FOR_INTERRUPTS frequently enough?
It's the latter. I don't think I can add a client-side timeout into
sqlsmith elegantly. IMHO it's better to write another test tool that
just re-runs the queries that were logged with a timeout by sqlsmith and
investigates their timeout-behavior more closely.
>> I do have some older numbers for line coverage from before the recent grammar extension:
>
> If you have a corpus of queries in a simple format it would be pretty
> convenient to add them in a regression test and then run make coverage
> to get html reports.
Hmm, I thought I found a workflow that would yield sqlsmith's coverage
without integrating it into the regession tests. This is what I did:
make install initdb /tmp/gcov pg_ctl -D /tmp/gcov start make installcheck pg_ctl -D /tmp/gcov stop make
coverage-clean pg_ctl -D /tmp/gcov start sqlsmith --target='dbname=regression' --max-queries=10000 pg_ctl -D
/tmp/gcovstop make coverage-html
It seems to yield a pure sqlsmith-only coverage report, as a "make
coverage-html" before the "make coverage-clean" yields a report with
much higher score. Maybe there are drawbacks to the workflow you are
suggesting? I just re-did it with the current sqlsmith code, and it's
up by 25% compared to the latest tested revision:
| revision | overall | parser |
|----------+---------+--------|
| a4c1989 | 26.0 | 20.4 |
| ee099e6 | 33.8 | 25.8 |
I also put the report here, in case someone wants to look at certain
details, or make suggestions into what directions to best extend the
grammar to increase coverage.
http://ansel.ydns.eu/~andreas/coverage/ http://ansel.ydns.eu/~andreas/gcov.tar.xz
> Did you publish the source already? I haven't been following all
> along, sorry if these are all answered questions.
It's not had a proper release yet, but the code is available via github
in all its rapid-prototypesque glory:
https://github.com/anse1/sqlsmith
regards,
Andreas