intarray: fix an edge case int32 overflow bug

/*

 * Comparison function for binary search in mcelem array.

 */

static int

compare_val_int4(const void *a, const void *b)

{

    int32       key = *(int32 *) a;

    const Datum *t = (const Datum *) b;

    return key - DatumGetInt32(*t);

}

#include <stdio.h>

#include <stdint.h>

#include <limits.h>

int main(void)

{

    int32_t a = INT32_MAX;

    int32_t b = -1;

    int32_t diff = a - b;  /* overflow */

    printf("a = %d\n", a);

    printf("b = %d\n", b);

    printf("a - b = %d\n", diff);

    return 0;

}

a = 2147483647

b = -1

a - b = -2147483648 <== INT32_MIN, but as a>b, we expect a positive value for bsearch comparator

The fix is straightforward: compare the key and *t without doing a potentially overflowing subtraction. Because no test covered this edge case, I added one. The test loads arrays [-2], [-1], and [INT32_MAX], ensuring -1 is the middle MCE value so the buggy comparator overflows on the first bsearch comparison and fail to find INT32_MAX. It then checks the EXPLAIN row estimation for a query on INT32_MAX. Before the fix, the lookup falls back to DEFAULT_EQ_SEL and EXPLAIN

reports rows=1; after the fix, it finds the MCE and reports the correct row estimation (the count of INT32_MAX rows).

--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/