On Sun, 1 Mar 2020 at 02:59, Justin AnyhowStep <anyhowstep@hotmail.com> wrote:
>
> > With this patch, all the examples originally posted return the correct
> > results (calculated with bc). I'd be interested to know how the OP
> > constructed these examples, and whether there were any that were off
> > by more than 1 ULP.
>
> > Yeah, that would be interesting.
>
> I didn't do anything fancy. I'm learning how to write an arbitrary precision math library.
> I couldn't come up with good test cases since so many numbers exist.
> So, I sanity-checked my code by testing many random inputs against another library and pg, and compared the results.
>
> Most of the time, the problem would be with my code.
> But I found a few cases where it looked like pg was off.
>
> I don't think I found any that were off by more than 1.
>
OK, thanks for looking, and thanks for the report.
I did some random testing of my own using 100 random numbers each with
100,000 digits to the left of the decimal point and 900 to the right,
and I was able to find one case where the error was 2 ULP, without the
patch. The overall results were as follows:
error | count
-------+-------
2 | 1
1 | 16
-1 | 20
0 | 63
(4 rows)
which is quite a high probability of being off by one. With the patch,
the results for the same 100 random numbers became:
error | count
-------+-------
1 | 1
-1 | 1
0 | 98
(3 rows)
So the off-by-two case was fixed (actually becoming zero), and the
probability of off-by-one errors was greatly reduced (although, as
expected, not entirely removed).
I think this is pretty much the worst case for our ln()
implementation, since it has both a large number of digits before the
decimal point, causing a lot of square roots, and a large number after
the decimal point, causing a large number of Taylor series terms to be
computed. Such cases are really only of academic interest, and for
most other cases, the probability of an off-by-one result should be
much lower.
Regards,
Dean