Обсуждение: gmpy adapter

Поиск
Список
Период
Сортировка

gmpy adapter

От
Daniel Popowich
Дата:
Hello, all,

I have a client app that reads in many values from a numeric column,
does some heavy computations, then writes the results back to another
numeric column.

Python's decimal.Decimal is SLOOOOoooow.  I'm trying to use gmpy.mpq
instead.  I have the adapter for reading the values from the database
working fine:

    numeric2mpq = lambda d,c: None if d is None else gmpy.mpq(d)
    MPQ = psycopg2.extensions.new_type((1700,), "MPQ", numeric2mpq)
    psycopg2.extensions.register_type(MPQ)

This is the adapter I'm using for the reverse (converting the mpq to a
string suitable for casting to numeric):

    def mpq2numeric(mpq):
    s = '%d::numeric/%d::numeric' % (mpq.numer(), mpq.denom())
    return psycopg2.extensions.AsIs(s)
    psycopg2.extensions.register_adapter(gmpy.mpq(0).__class__,
                     mpq2numeric)


While the adapter works, it seems less than optimal as it creates an
expression for the server to process, e.g:

    print psycopg2.extensions.adapt(gmpy.mpq('.333'))
    333::numeric/1000::numeric


Questions:

  1) Is there something I'm overlooking with gmpy that could make this
     much simpler?

  2) What other solutions do folk use for working around pythons slow,
     slow, slow Decimal?


Thanks,

Dan


Re: gmpy adapter

От
Daniele Varrazzo
Дата:
On Mon, Feb 28, 2011 at 4:56 PM, Daniel Popowich
<danielpopowich@gmail.com> wrote:

>  1) Is there something I'm overlooking with gmpy that could make this
>     much simpler?

Why don't you use mpf instead? Arbitrary precision float may map
better than rationals to the database decimals.

>  2) What other solutions do folk use for working around pythons slow,
>     slow, slow Decimal?

There are around C implementations compatible with Python decimals:
http://www.bytereef.org/mpdecimal/index.html for example.

Also notice that float64 has 53 bits of precision, 15 full decimal
digits. If you don't need more precision when you write data in the
database (even if you have used more during calculations) you may just
use the repr(float()) of your mpq to write into the database.

-- Daniele

Re: gmpy adapter

От
Daniele Varrazzo
Дата:
On Mon, Feb 28, 2011 at 11:07 PM, Daniele Varrazzo
<daniele.varrazzo@gmail.com> wrote:

> Also notice that float64 has 53 bits of precision, 15 full decimal
> digits. If you don't need more precision when you write data in the
> database (even if you have used more during calculations) you may just
> use the repr(float()) of your mpq to write into the database.

Or else, in the adapter you may use mpf to approximate the rational to
the precision you need.

    >>> gmpy.mpq(2,3)
    mpq(2,3)
    >>> gmpy.mpf(_, 100)
    mpf('6.666666666666666666666666666666666666667e-1',100)
    >>> str(_)
    '0.6666666666666666666666666666666666666667'

-- Daniele

Re: gmpy adapter

От
Daniele Varrazzo
Дата:
On Tue, Mar 1, 2011 at 5:03 PM, Daniel Popowich
<danielpopowich@gmail.com> wrote:

> Thanks for the ideas.  One thing I didn't specify in my original post
> is that the numeric columnss in my database are for monetary values,
> so I can't use binary floating point representations in my python
> code, else I risk inexactness and accumulated error in long
> computations.

Yes, of course: this is the normal reason to avoid floating point
numbers and go for fixed point.

> I need to use decimal floating point, e.g., decimal.Decimal (or
> cdecimal for a C implementation) or a rational object, like
> fractions.Fraction (or gmpy.mpq for a C impl).
>
> I could use mpq for internal computations then use mpf for the final
> conversion to the string I need for my adapter, but I want a single
> type I can use throughout my code without such concerns.
>
> I'm going to try cdecimal for now.

If cdecimal is good for you (i.e. if the decimal semantics is fine),
you may use mpf throughout your program instead of mpq. It is an exact
type (as exact as decimal of course: you can't represent all the
rationals) and likely to be more efficient than mpq. Plus, it maps
directly with the postgres decimal on I/O.

cdecimal may make your life easier instead if you have already working
code using the Python Decimal and you want to speed it up.

-- Daniele

Re: gmpy adapter

От
Daniel Popowich
Дата:
Daniele Varrazzo writes:
> On Mon, Feb 28, 2011 at 11:07 PM, Daniele Varrazzo
> <daniele.varrazzo@gmail.com> wrote:
>
> > Also notice that float64 has 53 bits of precision, 15 full decimal
> > digits. If you don't need more precision when you write data in the
> > database (even if you have used more during calculations) you may just
> > use the repr(float()) of your mpq to write into the database.
>
> Or else, in the adapter you may use mpf to approximate the rational to
> the precision you need.
>
>     >>> gmpy.mpq(2,3)
>     mpq(2,3)
>     >>> gmpy.mpf(_, 100)
>     mpf('6.666666666666666666666666666666666666667e-1',100)
>     >>> str(_)
>     '0.6666666666666666666666666666666666666667'

Daniele,

Thanks for the ideas.  One thing I didn't specify in my original post
is that the numeric columnss in my database are for monetary values,
so I can't use binary floating point representations in my python
code, else I risk inexactness and accumulated error in long
computations.

I need to use decimal floating point, e.g., decimal.Decimal (or
cdecimal for a C implementation) or a rational object, like
fractions.Fraction (or gmpy.mpq for a C impl).

I could use mpq for internal computations then use mpf for the final
conversion to the string I need for my adapter, but I want a single
type I can use throughout my code without such concerns.

I'm going to try cdecimal for now.

Thanks, again!

Cheers,

Dan

Re: gmpy adapter

От
Daniel Popowich
Дата:
Daniele Varrazzo writes:
> If cdecimal is good for you (i.e. if the decimal semantics is fine),
> you may use mpf throughout your program instead of mpq. It is an
> exact type (as exact as decimal of course: you can't represent all
> the rationals) and likely to be more efficient than mpq. Plus, it
> maps directly with the postgres decimal on I/O.
>

Really?  But it's binary floating point.  From the GMP manual,
http://gmplib.org/manual-4.3.2/Floating_002dpoint-Functions.html:

    The mantissa in stored in binary, as might be imagined from the
    fact precisions are expressed in bits. One consequence of this is
    that decimal fractions like 0.1 cannot be represented exactly. The
    same is true of plain IEEE double floats. This makes both highly
    unsuitable for calculations involving money or other values that
    should be exact decimal fractions. (Suitably scaled integers, or
    perhaps rationals, are better choices.)

> cdecimal may make your life easier instead if you have already
> working code using the Python Decimal and you want to speed it up.

First in importance is performance, but correctness cannot be
compromised.  Python Decimal semantics are fine for my application,
but performance is too slow.  A drop in replacement like cdecimal
makes life easier, but if a faster solution is available, I'll
exchange ease of use for performance.

Dan

Re: gmpy adapter

От
Daniele Varrazzo
Дата:
On Tue, Mar 1, 2011 at 5:39 PM, Daniel Popowich
<danielpopowich@gmail.com> wrote:
>
> Daniele Varrazzo writes:
>> If cdecimal is good for you (i.e. if the decimal semantics is fine),
>> you may use mpf throughout your program instead of mpq. It is an
>> exact type (as exact as decimal of course: you can't represent all
>> the rationals) and likely to be more efficient than mpq. Plus, it
>> maps directly with the postgres decimal on I/O.
>>
>
> Really?  But it's binary floating point.  From the GMP manual,
> http://gmplib.org/manual-4.3.2/Floating_002dpoint-Functions.html:


Uhm... I think you are right. I thought mpf was more similar to a
scaled integer.

-- Daniele