>BTW, would someone explain to me why using a float here will not fail catastrophically for inputs outside the range of
float?
Indeed, it will. And that's one of the reason I'm considering
advancing GiST API instead of advancing extensions.
It won't crash, but will produce terrible index, not suitable of
performing efficient queries.
GiST treats penalty this way:
/* disallow negative or NaN penalty */
if (isnan(penalty) || penalty < 0.0)
penalty = 0.0;
Any NaN is a "perfect fit".
Calculation of real (say float) penalty and choice of subtree with
smallest penalty is an idea from 92. Modern spatial index requires
more than just a float to compare subtrees.
Best regards, Andrey Borodin, Octonica & Ural Federal University.