Re: v17 vs v16 performance comparison

Hello Thomas.

01.08.2024 08:57, Thomas Munro wrote:
> On Thu, Aug 1, 2024 at 3:00 PM Alexander Lakhin <exclusion@gmail.com> wrote:
>> So it looks like q15 from TPC-DS is not the only query suffering from that
>> change.
> I'm going to try to set up a local repro to study these new cases.  If
> you have a write-up somewhere of how exactly you run that, that'd be
> useful.

I'm using this instrumentation (on my Ubuntu 22.04 workstation):
https://github.com/alexanderlaw/pg-mark.git
README.md can probably serve as a such write-up.

If you install all the prerequisites (some tests, including pg_tpcds,
require downloading additional resources; run-benchmarks.py will ask to
do that), there should be no problems with running benchmarks.

I just added two instances to config.xml:
         <instance id="pg-src-16" type="src" pg_version="16devel" git_branch="REL_16_STABLE" />
         <instance id="pg-src-17" type="src" pg_version="17devel" git_branch="REL_17_STABLE" />
and ran
1)
./prepare-instances.py -i pg-src-16 pg-src-17

2)
time ./run-benchmarks.py -i pg-src-16 pg-src-17 pg-src-16 pg-src-17 pg-src-17 pg-src-16
(it took 1045m55,215s on my machine so you may prefer to choose the single
benchmark (-b pg_tpcds or maybe s64da_tpcds))

3)
./analyze-benchmarks.py -i 'pg-src-17--.*' 'pg-src-16--.*'

All the upper-level commands to run benchmarks are contained in config.xml,
so you can just execute them separately, but my instrumentation eases
processing of the results by creating one unified benchmark-results.xml.

Please feel free to ask any questions or give your feedback.

Thank you for paying attention to this!

Best regards,
Alexander