Hmmm ... just when you thought it was safe to go back in the water ...
I was only looking closely at the "box" case earlier today, assuming
that the polygon code was set up identically. Well, it isn't. In
particular it appears that the poly_overleft and poly_overright
definitions are different from box's, which means that rtrees are
still broken for polygon searches.
I'm of the opinion that this is a flat-out bug and we should just
fix it, ie, change the operator definitions. The polygon definitions
aren't even self-consistent (overleft accepts equality and overright
doesn't).
poly_leftresult = polya->boundbox.high.x < polyb->boundbox.low.x;
poly_overleft:result = polya->boundbox.low.x <= polyb->boundbox.high.x;
poly_right:result = polya->boundbox.low.x > polyb->boundbox.high.x;
poly_overright:result = polya->boundbox.high.x > polyb->boundbox.low.x;
By analogy to the box case these should be
poly_overleft:result = polya->boundbox.high.x <= polyb->boundbox.high.x;
poly_overright:result = polya->boundbox.low.x >= polyb->boundbox.low.x;
regards, tom lane