Re: Possible rounding error of large float values?

# select 9223372036854775807 = 9223372036854775807::float;?column?
----------t
(1 row)

# select 9223372036854775807 = 9223372036854775807.0::float;?column?
----------t
(1 row)

# select 9223372036854775807 = 9223372036854775807.0000::numeric;?column?
----------t
(1 row)

This and the fact that it's still possible to find the row using the
original value would seem to indicate that the rounding is just a display
artifact..

----- Original Message ----- 
From: "Oliver Elphick" <olly@lfix.co.uk>
To: "Josh Berkus" <josh@agliodbs.com>
Cc: "Adam Lancaster" <alancaster@preventsys.com>; <pgsql-sql@postgresql.org>
Sent: Wednesday, August 25, 2004 5:21 AM
Subject: Re: [SQL] Possible rounding error of large float values?


> On Tue, 2004-08-24 at 20:52, Josh Berkus wrote:
> > Adam,
> >
> > > 9223372036854775807
> > >
> > > It gets selected out as:
> > >
> > > 9.22337203685478E18
> >
> > This is a property of FLOAT data types.  They round.
> >
> > > Which appears to be rounded. When we cast it to numeric type we get:
> > >
> > > 9223372036854780000
> > >
> > > Which also is rounded. It is still possible to find the row using the
> > > original value
> >
> > Hmmm ... is 15 digits the limit of NUMERIC?   It may be.
>
> It must be the limit of float.  Numeric can't produce any more than it
> was given and it was cast from float.
>
> bray=# select 653596708775675750507850507570708696432 ::numeric;
>                  numeric
> -----------------------------------------
>  653596708775675750507850507570708696432
> (1 row)
>
> -- 
> Oliver Elphick                                          olly@lfix.co.uk
> Isle of Wight                              http://www.lfix.co.uk/oliver
> GPG: 1024D/A54310EA  92C8 39E7 280E 3631 3F0E  1EC0 5664 7A2F A543 10EA
>                  ========================================
>      "I saw in the night visions, and, behold, one like the
>       Son of man came with the clouds of heaven, and came to
>       the Ancient of days, and they brought him near before
>       him. And there was given him dominion, and glory, and
>       a kingdom, that all people, nations, and languages,
>       should serve him; his dominion is an everlasting
>       dominion, which shall not pass away, and his kingdom
>       that which shall not be destroyed."
>                                     Daniel 7:13,14
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster