Re: MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

On Sat, Jun 22, 2013 at 03:46:49AM -0400, Stephen Frost wrote:
> * Noah Misch (noah@leadboat.com) wrote:
> > The next limit faced by sorts is
> > INT_MAX concurrent tuples in memory, which limits helpful work_mem to about
> > 150 GiB when sorting int4.
>
> That's frustratingly small. :(

I could appreciate a desire to remove that limit.  The way to do that is to
audit all uses of "int" variables in tuplesort.c and tuplestore.c, changing
them to Size where they can be used as indexes into the memtuples array.

Nonetheless, this new limit is about 50x the current limit; you need an
(unpartitioned) table of 2B+ rows to encounter it.  I'm happy with that.

> > !           if (memtupsize * grow_ratio < INT_MAX)
> > !                   newmemtupsize = (int) (memtupsize * grow_ratio);
> > !           else
> > !                   newmemtupsize = INT_MAX;
> >
> >             /* We won't make any further enlargement attempts */
> >             state->growmemtuples = false;
>
> I'm not a huge fan of moving directly to INT_MAX.  Are we confident that
> everything can handle that cleanly..?  I feel like it might be a bit
> safer to shy a bit short of INT_MAX (say, by 1K).  Perhaps that's overly
> paranoid, but there's an awful lot of callers and some loop which +2's
> and then overflows would suck, eg:

Where are you seeing "an awful lot of callers"?  The code that needs to be
correct with respect to the INT_MAX limit is all in tuplesort.c/tuplestore.c.
Consequently, I chose to verify that code rather than add a safety factor.  (I
did add an unrelated safety factor to repalloc_huge() itself.)

> Also, could this be used to support hashing larger sets..?  If we change
> NTUP_PER_BUCKET to one, we could end up wanting to create a hash table
> larger than INT_MAX since, with 8-byte pointers, that'd only be around
> 134M tuples.

The INT_MAX limit is an internal limit of tuplesort/tuplestore; other
consumers of the huge allocation APIs are only subject to that limit if they
find reasons to enforce it on themselves.  (Incidentally, the internal limit
in question is INT_MAX tuples, not INT_MAX bytes.)