During some development on encoding-related parts of postgres I stumbled over the new length-checking-code in common/hex.c/pg_hex_encode.
Differently when comparing it to all versions before the output-buffer-length is checked during encoding of every individual source byte.
Is always good to remove immutables from loops, if safe and secure.
This may impose quite a regression when using pg_dump on databases with many/big bytea or lo columns.
Decode has regression too.
Because all criteria to check buffer-length are known in advance of encoding (srclen and destlen) I propose doing the check only once before starting the while-loop.
Good.
Please find the attached patch for pg14 and master, older versions did not have this behavior.
I think that we can take one more step here.
pg_hex_decode can be improved too.
We can remove limitations from all functions that use hex functions.
byteaout from (varlena.c) has an artificial limitation to handle only MaxAllocSize length, but
nothing prevents her from being prepared for that limitation to be removed one day.
Tested on pg14-beta3, but applies also on master.
PS: This is my very first patch, I am in no way an experienced C-developer and don't have a smoothly running development environment or experience yet. (originally mostly dealing with postgres on Windows).
It seems good to me.
Please, take a look at the new version attached.
If possible can you review the tests if there is an overflow at
pg_hex_encode and pg_hex_decode functions?
regards,
Ranier Vilela