pgsql: Fix possible "invalid memory alloc request size" failure in node

Fix possible "invalid memory alloc request size" failure in nodeHash.c.

Limit the size of the hashtable pointer array to not more than
MaxAllocSize.  We've seen reports of failures due to this in HEAD/9.5,
and it seems possible in older branches as well.  The change in
NTUP_PER_BUCKET in 9.5 may have made the problem more likely, but
surely it didn't introduce it.

Tomas Vondra, slightly modified by me

Branch
------
REL9_1_STABLE

Details
-------
http://git.postgresql.org/pg/commitdiff/a8168fbcbbb3ed88c19ca2b648d737e8d90f37b1

Modified Files
--------------
src/backend/executor/nodeHash.c |    6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)