On 2018-Aug-04, Kohei KaiGai wrote:
> I could load the same data (544GB csv, 789GB heap) using COPY FROM successfully.
> When I reported the problem, rss usage of postgresql process increased
> about 10MB/s ratio, then OOM killer eliminated after a few hours.
OK, I think we can consider this particular bug closed, then.
> Now, it consumed about 60MB rss at the beginning of COPY FROM, and it
> grows up very slowly during the COPY FROM execution, then grew up to
> 250MB before completion.
> We may have another memory blocks which are not released during
> execution, however,
> I could not identify whether it is really memory leaking or not, and
> who's jobs.
Most likely, this is a different memory leak.
I sugges that one way to track this down is first figure out *which*
context is the one growing, which you can see by running
MemoryContextStats a few times and noting for meaningful differences.
Then we can try to narrow down what is allocating stuff in that context.
> It may be an idea to put a debug code that raises a notice when
> MemoryContext allocates more than the threshold.
I don't think this is really practical, because whatever the threshold
we set, there would be some corner-case scenario where the threshold is
legitimately crossed. And some memory leak scenarios that don't cross
any thresholds.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services