Re: Understanding memory usage

Поиск
Список
Период
Сортировка
От Damiano Albani
Тема Re: Understanding memory usage
Дата
Msg-id CAKys953M96Oj15f2fnVtFtQV1=91fwuNEq8KFPPEvXpk2cbAGg@mail.gmail.com
обсуждение исходный текст
Ответ на Re: Understanding memory usage  (Daniele Varrazzo <daniele.varrazzo@gmail.com>)
Ответы Re: Understanding memory usage
Список psycopg
On Wed, Oct 30, 2013 at 7:27 PM, Daniele Varrazzo <daniele.varrazzo@gmail.com> wrote:

What to you mean as "freed"? Have you deleted the cursor and made sure the gc reclaimed it? The cursor doesn't destroy the internal data until it is deleted or another query is run (because after fetchall() you can invoke scroll(0) and return it to Python again). And of course when the data returned by fetch() is released depends on the client usage.

By "freed", I mean doing like in the bug report #78:

    del data
    areadcur.close()
    acon.close()
    del areadcur
    del acon


After a big query you may see memory usage going down as soon as you execute "select 1 from false" because the result is replaced by a smaller one.

That's not the result that I get. Doing a query returning 2 millions rows followed by a "SELECT 1" has no effect on RSS memory usage in my case.

The only "problem" you may attribute to Psycopg is if you find an unbound usage of the memory. If you run some piece of code in a loop and see memory increasing linearly you have found a leak. Otherwise you can attribute the artefacts you see to the Python GC.

Indeed, there's no memory leak that I can see. But don't you find strange that Python / Psycopg memory management differs between 2 roughly equivalent query:
  • a query returning 20 rows × 10 MB each
  • a query returning 2 millions rows × 100 bytes each
As far as I could test, in my environment, they're clearly not equal in terms of side effects.
For the first, I can reclaim the memory after getting the results. For the second, I can't.

--
Damiano Albani

В списке psycopg по дате отправления:

Предыдущее
От: Daniele Varrazzo
Дата:
Сообщение: Re: Understanding memory usage
Следующее
От: Daniele Varrazzo
Дата:
Сообщение: Re: Understanding memory usage