Hi all,
When we build hash table for a hash join node etc., we split tuples into different hash buckets. Since tuples could not all be held in memory. Postgres splits each bucket into batches, only the current batch of bucket is in memory while other batches are written to disk.
During ExecHashTableInsert(), if the memory cost exceeds the operator allowed limit(hashtable->spaceAllowed), batches will be split on the fly by calling ExecHashIncreaseNumBatches().
In past, if data is distributed unevenly, the split of batch may failed(All the tuples falls into one split batch and the other batch is empty) Then Postgres will set hashtable->growEnable to false. And never expand batch number any more.
If tuples become diverse in future, spliting batch is still valuable and could avoid the current batch become too big and finally OOM.
To fix this, we introduce a penalty on hashtable->spaceAllowed, which is the threshold to determine whether to increase batch number.
If batch split failed, we increase the penalty instead of just turn off the growEnable flag.
Any comments?