Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Sep 22, 2011 at 12:24 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Now, having said that, I'm starting to wonder again why it's worth our
>> trouble to fool with encoding-specific incrementers. �The exactness of
>> the estimates seems unlikely to be improved very much by doing this.
> Well, so the problem is that the frequency with which the algorithm
> fails altogether seems to be disturbingly high for certain kinds of
> characters. I agree it might not be that important to get the
> absolutely best next string, but it does seem important not to fail
> outright. Kyotaro Horiguchi gives the example of UTF-8 characters
> ending with 0xbf.
[ thinks for a bit ... ] Yeah, it's certainly true that such a
character might be relatively small in the overall sort order. The
assumption underlying what we're doing now is that dropping the last
character and incrementing the next-to-last one instead isn't terribly
catastrophic from an estimation accuracy standpoint. I can see that
there are cases where that would fail to be true, but I'm not exactly
convinced that they're worse than all the other cases where we'll get
a poor estimate.
Anyway, I won't stand in the way of the patch as long as it's modified
to limit the number of values considered for any one character position
to something reasonably small.
regards, tom lane