Re: PostgreSQL 8.0.6 crash

On Thu, Feb 09, 2006 at 02:03:41PM -0500, Mark Woodward wrote:
> > "Mark Woodward" <pgsql@mohawksoft.com> writes:
> >> Again, regardless of OS used, hashagg will exceed "working memory" as
> >> defined in postgresql.conf.
> >
> > So?  If you've got OOM kill enabled, it can zap a process whether it's
> > strictly adhered to work_mem or not.  The OOM killer is entirely capable
> > of choosing a victim process whose memory footprint hasn't changed
> > materially since it started (eg, the postmaster).
> 
> Sorry, I must strongly disagree here. The postgresql.conf "working mem" is
> a VERY IMPORTANT setting, it is intended to limit the consumption of
> memory by the postgresql process. Often times PostgreSQL will work along

Actually, no, it's not designed for that at all.

> side other application servers on the same system, infact, may be a
> sub-part of application servers on the same system. (This is, in fact, how
> it is used on one of my site servers.)
> 
> Clearly, if the server will use 1000 times this number (Set for 1024K, but
> exceeds 1G) this is broken, and it may cause other systems to fail or
> perform very poorly.
> 
> If it is not something that can be fixed, it should be clearly documented.

work_mem (integer)
   Specifies the amount of memory to be used by internal sort   operations and hash tables before switching to
temporarydisk files.   The value is specified in kilobytes, and defaults to 1024 kilobytes   (1 MB). Note that for a
complexquery, several sort or hash   operations might be running in parallel; each one will be allowed to   use as much
memoryas this value specifies before it starts to put   data into temporary files. Also, several running sessions could
be  doing such operations concurrently. So the total memory used could   be many times the value of work_mem; it is
necessaryto keep this   fact in mind when choosing the value. Sort operations are used for   ORDER BY, DISTINCT, and
mergejoins. Hash tables are used in hash   joins, hash-based aggregation, and hash-based processing of IN   subqueries.


So it says right there that it's very easy to exceed work_mem by a very
large amount. Granted, this is a very painful problem to deal with and
will hopefully be changed at some point, but it's pretty clear as to how
this works.
-- 
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461