Re: rounding problems

Поиск
Список
Период
Сортировка
От Sam Mason
Тема Re: rounding problems
Дата
Msg-id 20080514162630.GG1657@frubble.xen.chris-lamb.co.uk
обсуждение исходный текст
Ответ на Re: rounding problems  (Justin <justin@emproshunts.com>)
Ответы Re: rounding problems
Список pgsql-general
On Wed, May 14, 2008 at 11:47:52AM -0400, Justin wrote:
> I have forgotten how much i hate C++

What we're talking about doesn't have much to do with C++, it's floating
point maths in general.

> Its not doing what you say it would but it did do other odd ball
> things.  I miss my foxpro :-(.

What does foxpro use for storing numbers? or is it just that you never
pushed it hard enough for the abstractions to show through.

> Plus its not holding 15 precision points

after changing the output to be:

  printf("%.10f %.10f\n", d, d-c);

I get:

  100000000.0999999940 0.0999999940
  100000000.0100000054 0.0100000054
  100000000.0010000020 0.0010000020
  100000000.0001000017 0.0001000017
  100000000.0000099987 0.0000099987
  100000000.0000009984 0.0000009984
  100000000.0000001043 0.0000001043
  100000000.0000000149 0.0000000149
  100000000.0000000000 0.0000000000

Which looks reasonable.  Remember that floating point numbers store
their state in base two, not base ten.  All of those numbers look good
to 15 decimal digits.


  Sam

В списке pgsql-general по дате отправления:

Предыдущее
От: Decibel!
Дата:
Сообщение: Re: Request for Materialized Views
Следующее
От: Glyn Astill
Дата:
Сообщение: Re: postgres crash when select a record