Re: PG_DUMP very slow because of STDOUT ??

Hi Craig,

Now I did the test with top too. Free RAM is around 900. And there doesn't seem to be other processes eating memory
away(looked at it with top/htop). The other procs having more RAM did have it before (mostly some postgres processes),
anddon't grow it at an exporbitant rate. One coul see with free, that most of the time, when it is eaten away, it goes
directlyto cache. I also observe, that the lower "free" is, the slower COPY-to-STDOUT becomes ... and linearly, the
numberof times one can see congestion_wait in PS grows. So, with more "free" available (even when it is not eaten away
anymore...) the more responsive COPY-to-STDOUT. 

Well, I have now also rerun a few times my plain COPY-to-FILE test. That one seems to behave similarly with the RAM
(whenmonitored via "free -m") ... but it never gets into congestion. So, maybe, just maybe ... could this be some kind
ofrace condition? I mean, the file cache going for RAM (or the memory mapped file ???) and the SOCKET going for some
memorytoo, and then they somehow overwhelm the memory manager (which shouldn't happen). ... Well, this might be a very
wildspeculation, but still. 

I now also looked at the stack, before the congestion_wait. There are some "interesting" calls leading to it. Like
"shrink_list","zone_reclaim" ... after the call of "get_page_from_freelist". Sounds like some heavy memory juggling
goingon (to get new free pages) ... What ever the reason is for this. 

Andras Fabian


-----Ursprüngliche Nachricht-----
Von: Craig Ringer [mailto:craig@postnewspapers.com.au]
Gesendet: Dienstag, 13. Juli 2010 14:11
An: Andras Fabian
Cc: pgsql-general@postgresql.org
Betreff: Re: AW: AW: AW: AW: [GENERAL] PG_DUMP very slow because of STDOUT ??

On 13/07/10 18:57, Andras Fabian wrote:
> OK, so here I should - maybe - look around the sockets. Hmm. Well, in the case of my experiments we are talking about
Unixsockets, as I am only connecting locally to the server (not real networking involved). Are there any ideas, where
sucha Unix Socket could impose such extreme buffering ??? And can/could I control its behavior at all?? (or would it be
athing, which can only be controlled from C-Code ... which would fall back to PostgreSQL as the initiator). 

Confirm your issue is actually the socket between client and server first.

Are you sure the client isn't buffering the data? Or a pager like
"less"? Or terminal scrollback?

Run "top" and sort by memory use by pressing Shift-M. Run your query.
Look at the output and see if anything grows lots.

You really need to start looking at if this is memory consumption, and
what's using it. Don't get fixated on one answer until you've excluded
other possibilities.

--
Craig Ringer