Обсуждение: pgbench is badly broken since July

I have just realized that the recent patches in pgbench have altered its
behavior in a way that destroys reproducibility of results --- I'm
seeing reported TPS numbers about twice what they were before that.
I'd love to say we did something in the past month that made the backend
2X faster, but sadly, tain't so :-(

I've only done limited investigation but what seems to be happening
is that the "-s" scale factor is no longer being applied to adjust the
range of the random variables, leading to only part of the database
being touched.  For example, with -s = 10, the older code updates all 10
"branches" rows but CVS HEAD only seems to touch bid = 1.

I see that the code still thinks it is scaling the numbers, but the
mechanism for doing that has changed drastically, and I suspect that's
the problem.  I assume there's some garden-variety bug in there.

At a more fundamental level, the recent patches seem to mean that
pg_dump is doing noticeably more work to issue its queries than it was
before, and I wonder whether this isn't going to destroy cross-version
reproducibility in its own way.  Have you done any timing studies to
verify that the new and old code have comparable performance?  I don't
mind enhancing pgbench's functionality for non-default tests, but I
think it's important that the default case remain comparable over time.
If there is a noticeable speed difference then I'd vote for going back
to the klugy old code for adjusting the default script's values.

And at the nitpicking level, is ":tps" a good name for the variable that
reflects the scaling factor?  It seems awfully easy to confuse that with
the TPS numbers that pgbench reports.  Perhaps ":scale" or some such
would be better.
        regards, tom lane

I'll look into this.
--
Tatsuo Ishii
SRA OSS, Inc. Japan

> I have just realized that the recent patches in pgbench have altered its
> behavior in a way that destroys reproducibility of results --- I'm
> seeing reported TPS numbers about twice what they were before that.
> I'd love to say we did something in the past month that made the backend
> 2X faster, but sadly, tain't so :-(
> 
> I've only done limited investigation but what seems to be happening
> is that the "-s" scale factor is no longer being applied to adjust the
> range of the random variables, leading to only part of the database
> being touched.  For example, with -s = 10, the older code updates all 10
> "branches" rows but CVS HEAD only seems to touch bid = 1.
> 
> I see that the code still thinks it is scaling the numbers, but the
> mechanism for doing that has changed drastically, and I suspect that's
> the problem.  I assume there's some garden-variety bug in there.
> 
> At a more fundamental level, the recent patches seem to mean that
> pg_dump is doing noticeably more work to issue its queries than it was
> before, and I wonder whether this isn't going to destroy cross-version
> reproducibility in its own way.  Have you done any timing studies to
> verify that the new and old code have comparable performance?  I don't
> mind enhancing pgbench's functionality for non-default tests, but I
> think it's important that the default case remain comparable over time.
> If there is a noticeable speed difference then I'd vote for going back
> to the klugy old code for adjusting the default script's values.
> 
> And at the nitpicking level, is ":tps" a good name for the variable that
> reflects the scaling factor?  It seems awfully easy to confuse that with
> the TPS numbers that pgbench reports.  Perhaps ":scale" or some such
> would be better.
> 
>             regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
> 
>                http://www.postgresql.org/docs/faq
> 

> I have just realized that the recent patches in pgbench have altered its
> behavior in a way that destroys reproducibility of results --- I'm
> seeing reported TPS numbers about twice what they were before that.
> I'd love to say we did something in the past month that made the backend
> 2X faster, but sadly, tain't so :-(
> 
> I've only done limited investigation but what seems to be happening
> is that the "-s" scale factor is no longer being applied to adjust the
> range of the random variables, leading to only part of the database
> being touched.  For example, with -s = 10, the older code updates all 10
> "branches" rows but CVS HEAD only seems to touch bid = 1.

As far as I concern the bug is there, but in a different way what you
said. It seems "-s" works. Problem is, pgbench does not get the
default scaling factor (that is same as the number of rows in branches
table), instead alway uses the default scaling factor 1. I have fixed
the problem in CVS HEAD.

> I see that the code still thinks it is scaling the numbers, but the
> mechanism for doing that has changed drastically, and I suspect that's
> the problem.  I assume there's some garden-variety bug in there.
> 
> At a more fundamental level, the recent patches seem to mean that
> pg_dump is doing noticeably more work to issue its queries than it was
> before, and I wonder whether this isn't going to destroy cross-version
> reproducibility in its own way.  Have you done any timing studies to
> verify that the new and old code have comparable performance?  I don't
> mind enhancing pgbench's functionality for non-default tests, but I
> think it's important that the default case remain comparable over time.
> If there is a noticeable speed difference then I'd vote for going back
> to the klugy old code for adjusting the default script's values.

Let me do further performance testing.

> And at the nitpicking level, is ":tps" a good name for the variable that
> reflects the scaling factor?  It seems awfully easy to confuse that with
> the TPS numbers that pgbench reports.  Perhaps ":scale" or some such
> would be better.

I replaced all occurenes of "tps" to scale.
--
Tatsuo Ishii
SRA OSS, Inc. Japan