pgsql: Protect GIST logic that assumes penalty values can't be negative

Protect GIST logic that assumes penalty values can't be negative.

Apparently sane-looking penalty code might return small negative values,
for example because of roundoff error.  This will confuse places like
gistchoose().  Prevent problems by clamping negative penalty values to
zero.  (Just to be really sure, I also made it force NaNs to zero.)
Back-patch to all supported branches.

Alexander Korotkov

Branch
------
REL8_4_STABLE

Details
-------
http://git.postgresql.org/pg/commitdiff/8057b7554bde98d887f02f5c0c7aeca01c8b52c9

Modified Files
--------------
doc/src/sgml/gist.sgml             |    2 ++
src/backend/access/gist/gistutil.c |   14 +++++++++++---
2 files changed, 13 insertions(+), 3 deletions(-)