Re: numeric precision when raising one numeric to another.

Alvaro Herrera wrote:
> On Fri, May 20, 2005 at 01:03:08PM -0400, Tom Lane wrote:
>
> Oh, and while at it, it would be nice to solve the modulo bug that still
> lurks there:
>
> alvherre=# select 12345678901234567890 % 123;
>  ?column?
> ----------
>       -45
> (1 fila)
>
> alvherre=# select 12345678901234567890 % 123::numeric(4,1);
>  ?column?
> ----------
>      78.0
> (1 fila)
>
> alvherre=# select 12345678901234567890 % 123::numeric(3,0);
>  ?column?
> ----------
>       -45
> (1 fila)
>
> alvherre=# select version();
>                                            version
> ----------------------------------------------------------------------------------------------
>  PostgreSQL 8.1devel on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.3.6 (Debian 1:3.3.6-4)
> (1 fila)

I poked around on this one and found this in the comments in
numeric::mod_var():

    /* ---------
     * We do this using the equation
     *      mod(x,y) = x - trunc(x/y)*y
     * We set rscale the same way numeric_div and numeric_mul do
     * to get the right answer from the equation.  The final result,
     * however, need not be displayed to more precision than the inputs.
     * ----------
     */

so I tried it:

    test=> select 12345678901234567890 % 123;
     ?column?
    ----------
          -45
    (1 row)

    test=> select 12345678901234567890 / 123;
          ?column?
    --------------------
     100371373180768845
    (1 row)

    test=> select 100371373180768845::numeric * 123::numeric;
           ?column?
    ----------------------
     12345678901234567935
    (1 row)

    test=> select 12345678901234567890 - 12345678901234567935;
     ?column?
    ----------
          -45
    (1 row)

and I was quite surprised at the result.  Basically, it looks like the
division is rounding _up_ the next integer on the /123 division, and
that is causing the modulo error.  In fact, should the /123 round up
with numeric?  I think there is an assumption in our code that div_var()
will not round up, but in fact it does in this case.

Here is 'calc' showing the same calculation:

    > 12345678901234567890 % 123
            78
    > 12345678901234567890 / 123
            ~100371373180768844.63414634146341463414
    > 100371373180768845 * 123

          ^^^^^^^^^^^^^^^^^^ rounded up by me

            12345678901234567935
    > 12345678901234567890 - 12345678901234567935
            -45

and here is 'bc' doing integer division:

    12345678901234567890 / 123
    100371373180768844
    100371373180768844 * 123
    12345678901234567812
    12345678901234567890 - 12345678901234567812
    78

This is why 123::numeric(4,1) fixes it because the division returns on
digit that is truncated, rather than rounding up to the next whole
number.

I am not sure how to fix this.  Adding extra scale to the division would
help, but if the division returned .999 and we had a scale of 2, it
would still round up and the truncate would not see it.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073