Re: Improve the performance of Unicode Normalization Forms.

Hi, Jeff, hackers!

As promised, refactoring the C code for Unicode Normalization Forms.

In general terms, here's what has changed:
1. Recursion has been removed; now data is generated using
     a Perl script.
2. Memory is no longer allocated for uint32 for the entire size,
     but uint8 is allocated for the entire size for the CCC cache, which
     boosts performance significantly.
3. The code for the unicode_normalize() function has been completely
     rewritten.

I am confident that we have achieved excellent results.

Jeff's test:
Without patch:
     Normalization from NFC to  NFD with  PG: 009.121
     Normalization from NFC to NFKD with  PG: 009.048
     Normalization from NFD to  NFC with  PG: 014.525
     Normalization from NFD to NFKC with  PG: 014.380

Whith patch:
     Normalization from NFC to  NFD with  PG: 001.580
     Normalization from NFC to NFKD with  PG: 001.634
     Normalization from NFD to  NFC with  PG: 002.979
     Normalization from NFD to NFKC with  PG: 003.050

Test with ICU (with path and ICU):
     Normalization from NFC to  NFD with  PG: 001.580
     Normalization from NFC to  NFD with ICU: 001.880
     Normalization from NFC to NFKD with  PG: 001.634
     Normalization from NFC to NFKD with ICU: 001.857

     Normalization from NFD to  NFC with  PG: 002.979
     Normalization from NFD to  NFC with ICU: 001.144
     Normalization from NFD to NFKC with  PG: 003.050
     Normalization from NFD to NFKC with ICU: 001.260

pgbench:
The files were sent via pgbench. The files contain all code points that
need to be normalized.

NFC:
     Patch: tps = 9701.568161
     Without: tps = 6820.828104

NFD:
     Patch: tps = 2707.155148
     Without: tps = 1745.949174

NFKC:
     Patch: tps = 9893.952804
     Without: tps = 6697.358888

NFKD:
     Patch: tps = 2580.785909
     Without: tps = 1521.058417

To ensure fairness in testing with ICU, I corrected Jeff's patch;
we calculate the size of the final buffer, and I placed ICU in
the same position.

I'm talking about:
Get size:
     length = unorm_normalize(u_input, -1, form, 0, NULL, 0, &status);
Normalize:
     unorm_normalize(u_input, -1, form, 0, u_result, length, &status);

Otherwise, it turned out that we were giving the ICU some huge buffer,
and it was writing to it.
And we ourselves calculate what buffer we need.


-- 
Regards,
Alexander Borisov