Okay, I've been able to reproduce the problem here. Looks like the
eqjoinsel math is not wrong, exactly, but small roundoff errors are
causing the logic to do unreasonable things. I think
get_att_numdistinct needs to round its result to an integer, and
probably there needs to be some clamping of computed probabilities to
the 0..1 range (otherfreq1 is coming out about -4.4703483581542969e-08
in this example, which should be clamped to 0).
Will have a fix late tonight or tomorrow.
Thanks for the example case!
regards, tom lane