Re: libpq - lack of support to set the fetch size

Поиск
Список
Период
Сортировка
От Jeff Janes
Тема Re: libpq - lack of support to set the fetch size
Дата
Msg-id CAMkU=1zhd9CDKNVm3yQmwTJxRfefZ8-DB-jjWAaFQj3GOTs+rg@mail.gmail.com
обсуждение исходный текст
Ответ на libpq - lack of support to set the fetch size  (matshyeq <matshyeq@gmail.com>)
Ответы Re: libpq - lack of support to set the fetch size  ("Greg Sabino Mullane" <greg@turnstep.com>)
Список pgsql-general
On Sun, Mar 9, 2014 at 6:43 AM, matshyeq <matshyeq@gmail.com> wrote:
Hello,

I've found an issue when tried to implement fetching rows from big table (2mln rows) in my app.
Basically I don't find an elegant and easy way (other than always use cursors) to limit the number of rows returned.
This causes my application to break due to the excessive memory consumption.

I'm using Perl and DBD::Pg library but contacted maintainer who actually pointed out this is an issue that goes much deeper (libpq):

"Unfortunately, this is a limitation in the underlying driver (libpq) rather than DBD::Pg itself. There have been talks over the years of supporting this, but nothing concrete yet. Your best bet would be to ask about this on the Postgres lists"

I don't think this is correct.  First, DBD::Pg could get tricky and automatically wrap your query in a cursor and then fetch from the cursor behind the scenes.  I believe that this is what Python's module does for you in some modes.  Second, the feature needed to do this without even using a cursor was added 1.5 years ago (PQsetSingleRowMode).  The DBD::Pg was just not taught how to use it yet.

The first strategy could probably be done purely in Perl, the second would require changes to the C parts of DBD::Pg.

Of course just because it can be implemented in DBD::Pg doesn't mean anyone has an obligation to do it.  You could speed that along by contributing the code yourself.  But I would say the ball is firmly in DBD::Pg's court.


Cheers,

Jeff

В списке pgsql-general по дате отправления:

Предыдущее
От: Thom Brown
Дата:
Сообщение: Re: 9.1.11 - many backends in "semtimedop" syscall
Следующее
От: Tim Kane
Дата:
Сообщение: Re: Playing with 9.4devel - unnest