Re: [GENERAL] Gripe: bytea_output default => data corruption
Re: [GENERAL] Gripe: bytea_output default => data corruption
От:
Bruce Momjian <bruce@momjian.us>
Дата:
ljb wrote:
> tgl@sss.pgh.pa.us wrote:
> > I think we should simply remove the description of *how* the escaping is
> > performed, and state only that the function produces a suitably escaped
> > literal string. Anything else is not future-proof, and could someday
> > break the way this wording did.
>
> Perhaps it would be best to remove escaping details here. But the
> description of PQescapeBytea() might need to be rewritten, too. Without
> describing exactly what PQescapeByteaConn() does, it is hard to understand
> what PQescapeBytea() does not do, and why it therefore "might give the
> wrong results".
>
> I think the actual function behavior should be documented somewhere. Even
> though it might change again.
Based on this report, I have created the attached documentation patch
which clarifies the libpq behavior for escaping bytea. I am planning to
backpatch this to 9.0 as well.
--
Bruce Momjian http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 7b70970..26f5140 100644
*** a/doc/src/sgml/libpq.sgml
--- b/doc/src/sgml/libpq.sgml
*************** unsigned char *PQescapeByteaConn(PGconn
*** 3419,3434 ****
! Certain byte values must be escaped (but all
! byte values can be escaped) when used as part
! of a bytea literal in an SQL
! statement. In general, to escape a byte, it is converted into the
! three digit octal number equal to the octet value, and preceded by
! usually two backslashes. The single quote ('</>) and backslash
! (\</>) characters have special alternative escape
! sequences. See for more
! information. PQescapeByteaConn performs this
! operation, escaping only the minimally required bytes.
--- 3419,3429 ----
! Certain byte values must be escaped when used as part of a
! bytea literal in an SQL statement.
! PQescapeByteaConn escapes such bytes using
! either hex encoding or backslash escaping. See for more information.
*************** unsigned char *PQescapeBytea(const unsig
*** 3486,3495 ****
The only difference from PQescapeByteaConn</> is that
PQescapeBytea</> does not take a PGconn</>
parameter. Because of this, it cannot adjust its behavior
! depending on the connection properties (in particular, whether
! standard-conforming strings are enabled) and therefore
! it might give the wrong results</>. Also, it has no
! way to return an error message on failure.
--- 3481,3490 ----
The only difference from PQescapeByteaConn</> is that
PQescapeBytea</> does not take a PGconn</>
parameter. Because of this, it cannot adjust its behavior
! depending on the connection properties (e.g.
! standard_conforming_strings</>, bytea_output)
! and therefore might give the wrong results</>. Also,
! it has no way to return an error message on failure.
diff --git a/doc/src/sgml/release-9.0.sgml b/doc/src/sgml/release-9.0.sgml
index 2288f1b..c43f142 100644
*** a/doc/src/sgml/release-9.0.sgml
--- b/doc/src/sgml/release-9.0.sgml
***************
*** 2342,2348 ****
whether hex or traditional format is used for bytea</>
output. Libpq's PQescapeByteaConn()</> function automatically
uses the hex format when connected to PostgreSQL</> 9.0
! or newer servers.
--- 2342,2349 ----
whether hex or traditional format is used for bytea</>
output. Libpq's PQescapeByteaConn()</> function automatically
uses the hex format when connected to PostgreSQL</> 9.0
! or newer servers. However, pre-9.0 libpq versions will not
! correctly process hex format from newer servers.
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 7b70970..26f5140 100644
*** a/doc/src/sgml/libpq.sgml
--- b/doc/src/sgml/libpq.sgml
*************** unsigned char *PQescapeByteaConn(PGconn
*** 3419,3434 ****
! Certain byte values must be escaped (but all
! byte values can be escaped) when used as part
! of a bytea literal in an SQL
! statement. In general, to escape a byte, it is converted into the
! three digit octal number equal to the octet value, and preceded by
! usually two backslashes. The single quote ('</>) and backslash
! (\</>) characters have special alternative escape
! sequences. See for more
! information. PQescapeByteaConn performs this
! operation, escaping only the minimally required bytes.
--- 3419,3429 ----
! Certain byte values must be escaped when used as part of a
! bytea literal in an SQL statement.
! PQescapeByteaConn escapes such bytes using
! either hex encoding or backslash escaping. See for more information.
*************** unsigned char *PQescapeBytea(const unsig
*** 3486,3495 ****
The only difference from PQescapeByteaConn</> is that
PQescapeBytea</> does not take a PGconn</>
parameter. Because of this, it cannot adjust its behavior
! depending on the connection properties (in particular, whether
! standard-conforming strings are enabled) and therefore
! it might give the wrong results</>. Also, it has no
! way to return an error message on failure.
--- 3481,3490 ----
The only difference from PQescapeByteaConn</> is that
PQescapeBytea</> does not take a PGconn</>
parameter. Because of this, it cannot adjust its behavior
! depending on the connection properties (e.g.
! standard_conforming_strings</>, bytea_output)
! and therefore might give the wrong results</>. Also,
! it has no way to return an error message on failure.
Re: [GENERAL] Gripe: bytea_output default => data corruption
От:
Bruce Momjian <bruce@momjian.us>
Дата:
Bruce Momjian wrote:
> ljb wrote:
> > tgl@sss.pgh.pa.us wrote:
> > > I think we should simply remove the description of *how* the escaping is
> > > performed, and state only that the function produces a suitably escaped
> > > literal string. Anything else is not future-proof, and could someday
> > > break the way this wording did.
> >
> > Perhaps it would be best to remove escaping details here. But the
> > description of PQescapeBytea() might need to be rewritten, too. Without
> > describing exactly what PQescapeByteaConn() does, it is hard to understand
> > what PQescapeBytea() does not do, and why it therefore "might give the
> > wrong results".
> >
> > I think the actual function behavior should be documented somewhere. Even
> > though it might change again.
>
> Based on this report, I have created the attached documentation patch
> which clarifies the libpq behavior for escaping bytea. I am planning to
> backpatch this to 9.0 as well.
My aplogies on the double-attachment. I am now attaching the right
verison.
--
Bruce Momjian http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 7b70970..26f5140 100644
*** a/doc/src/sgml/libpq.sgml
--- b/doc/src/sgml/libpq.sgml
*************** unsigned char *PQescapeByteaConn(PGconn
*** 3419,3434 ****
! Certain byte values must be escaped (but all
! byte values can be escaped) when used as part
! of a bytea literal in an SQL
! statement. In general, to escape a byte, it is converted into the
! three digit octal number equal to the octet value, and preceded by
! usually two backslashes. The single quote ('</>) and backslash
! (\</>) characters have special alternative escape
! sequences. See for more
! information. PQescapeByteaConn performs this
! operation, escaping only the minimally required bytes.
--- 3419,3429 ----
! Certain byte values must be escaped when used as part of a
! bytea literal in an SQL statement.
! PQescapeByteaConn escapes such bytes using
! either hex encoding or backslash escaping. See for more information.
*************** unsigned char *PQescapeBytea(const unsig
*** 3486,3495 ****
The only difference from PQescapeByteaConn</> is that
PQescapeBytea</> does not take a PGconn</>
parameter. Because of this, it cannot adjust its behavior
! depending on the connection properties (in particular, whether
! standard-conforming strings are enabled) and therefore
! it might give the wrong results</>. Also, it has no
! way to return an error message on failure.
--- 3481,3490 ----
The only difference from PQescapeByteaConn</> is that
PQescapeBytea</> does not take a PGconn</>
parameter. Because of this, it cannot adjust its behavior
! depending on the connection properties (e.g.
! standard_conforming_strings</>, bytea_output)
! and therefore might give the wrong results</>. Also,
! it has no way to return an error message on failure.
diff --git a/doc/src/sgml/release-9.0.sgml b/doc/src/sgml/release-9.0.sgml
index 2288f1b..c43f142 100644
*** a/doc/src/sgml/release-9.0.sgml
--- b/doc/src/sgml/release-9.0.sgml
***************
*** 2342,2348 ****
whether hex or traditional format is used for bytea</>
output. Libpq's PQescapeByteaConn()</> function automatically
uses the hex format when connected to PostgreSQL</> 9.0
! or newer servers.
--- 2342,2349 ----
whether hex or traditional format is used for bytea</>
output. Libpq's PQescapeByteaConn()</> function automatically
uses the hex format when connected to PostgreSQL</> 9.0
! or newer servers. However, pre-9.0 libpq versions will not
! correctly process hex format from newer servers.
Re: [GENERAL] Gripe: bytea_output default => data corruption
От:
ljb <ljb9832@pobox.com>
Дата:
bruce@momjian.us wrote: >... >> Based on this report, I have created the attached documentation patch >> which clarifies the libpq behavior for escaping bytea. I am planning to >> backpatch this to 9.0 as well. This change says PQescapeBytea is unable to adjust its behavior based on bytea_output, implying that PQescapeByteaConn does adjust its behavior based on byte_output. Wrong! Neither one knows or cares about the bytea_output parameter, which is solely for the backend to tell it how to present bytea data to the front-end. Not for the front-ends at all. (See why we need this behavior documented?) Non-use of 'standard_conforming_strings' is also wrong, in the current (pre-patched) docs. PQescapeBytea does adjust its behavior for standard_conforming_strings, but only for the single-connection case. See similar text for PQescapeString. Regarding the first paragraph changed, it seems off to me. "PQescapeByteConn escapes such bytes using either hex encoding..." tells me that when hex encoding is used, PQescapeByte encodes only bytes that need escaping. Not true - hex encoding encodes (not escapes) every byte.
Re: [GENERAL] Gripe: bytea_output default => data corruption
От:
Bruce Momjian <bruce@momjian.us>
Дата:
ljb wrote:
> bruce@momjian.us wrote:
> >...
> >> Based on this report, I have created the attached documentation patch
> >> which clarifies the libpq behavior for escaping bytea. I am planning to
> >> backpatch this to 9.0 as well.
>
> This change says PQescapeBytea is unable to adjust its behavior based on
> bytea_output, implying that PQescapeByteaConn does adjust its behavior
> based on byte_output. Wrong! Neither one knows or cares about the bytea_output
> parameter, which is solely for the backend to tell it how to present bytea
> data to the front-end. Not for the front-ends at all.
Good point, sorry.
> (See why we need this behavior documented?)
Yep!
> Non-use of 'standard_conforming_strings' is also wrong, in the current
> (pre-patched) docs. PQescapeBytea does adjust its behavior for
> standard_conforming_strings, but only for the single-connection case. See
> similar text for PQescapeString.
Yep.
> Regarding the first paragraph changed, it seems off to me. "PQescapeByteConn
> escapes such bytes using either hex encoding..." tells me that when hex
> encoding is used, PQescapeByte encodes only bytes that need escaping. Not
> true - hex encoding encodes (not escapes) every byte.
OK, great feedback. I made the adjustments you suggested and tried to
pattern it after PQescapeStringConn, with adjustments where appropriate.
Here is an updated patch, again to be backpatched to 9.0. There were
obviously big mistakes in libpq docs and I am embarrassed I waited so
long to fix this; my apologies.
--
Bruce Momjian http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index e78d708..56b72a7 100644
*** a/doc/src/sgml/libpq.sgml
--- b/doc/src/sgml/libpq.sgml
*************** size_t PQescapeStringConn(PGconn *conn,
*** 3371,3385 ****
size_t PQescapeString (char *to, const char *from, size_t length);
! PQescapeString</> is an older, deprecated version of
! PQescapeStringConn</>; the difference is that it does
! not take conn</> or error</> parameters.
Because of this, it cannot adjust its behavior depending on the
connection properties (such as character encoding) and therefore
it might give the wrong results</>. Also, it has no way
--- 3371,3387 ----
+ PQescapeString</> is an older, deprecated version of
+ PQescapeStringConn</>.
size_t PQescapeString (char *to, const char *from, size_t length);
! The only difference from PQescapeStringConn</> is that
! PQescapeString</> does not take a PGconn</>
! or error</> parameters.
Because of this, it cannot adjust its behavior depending on the
connection properties (such as character encoding) and therefore
it might give the wrong results</>. Also, it has no way
*************** size_t PQescapeString (char *to, const c
*** 3387,3393 ****
! PQescapeString</> can be used safely in single-threaded
client programs that work with only one PostgreSQL</>
connection at a time (in this case it can find out what it needs to
know behind the scenes</>). In other contexts it is a security
--- 3389,3395 ----
! PQescapeString</> can be used safely in
client programs that work with only one PostgreSQL</>
connection at a time (in this case it can find out what it needs to
know behind the scenes</>). In other contexts it is a security
*************** unsigned char *PQescapeByteaConn(PGconn
*** 3419,3434 ****
! Certain byte values must be escaped (but all
! byte values can be escaped) when used as part
! of a bytea literal in an SQL
! statement. In general, to escape a byte, it is converted into the
! three digit octal number equal to the octet value, and preceded by
! usually two backslashes. The single quote ('</>) and backslash
! (\</>) characters have special alternative escape
! sequences. See for more
! information. PQescapeByteaConn performs this
! operation, escaping only the minimally required bytes.
--- 3421,3431 ----
! Certain byte values must be escaped when used as part of a
! bytea literal in an SQL statement.
! PQescapeByteaConn escapes bytes using
! either hex encoding or backslash escaping. See for more information.
*************** unsigned char *PQescapeBytea(const unsig
*** 3485,3504 ****
The only difference from PQescapeByteaConn</> is that
PQescapeBytea</> does not take a PGconn</>
! parameter. Because of this, it cannot adjust its behavior
! depending on the connection properties (in particular, whether
! standard-conforming strings are enabled) and therefore
! it might give the wrong results</>. Also, it has no
! way to return an error message on failure.
!
!
!
! PQescapeBytea</> can be used safely in single-threaded
! client programs that work with only one PostgreSQL</>
! connection at a time (in this case it can find out what it needs
! to know behind the scenes</>). In other contexts it is
! a security hazard and should be avoided in favor of
! PQescapeByteaConn</>.
--- 3482,3495 ----
The only difference from PQescapeByteaConn</> is that
PQescapeBytea</> does not take a PGconn</>
! or error</> parameters. Because of this,
! PQescapeBytea</> can only be used safely in client
! programs that use a single PostgreSQL</> connection
! at a time (in this case it can find out what it needs to know
! behind the scenes</>). It might give the
! wrong results</> if used in programs that use multiple database
! connections (use PQescapeByteaConn</> in such cases).
! Also, it has no way to report error conditions.
diff --git a/doc/src/sgml/release-9.0.sgml b/doc/src/sgml/release-9.0.sgml
index 2288f1b..c43f142 100644
*** a/doc/src/sgml/release-9.0.sgml
--- b/doc/src/sgml/release-9.0.sgml
***************
*** 2342,2348 ****
whether hex or traditional format is used for bytea</>
output. Libpq's PQescapeByteaConn()</> function automatically
uses the hex format when connected to PostgreSQL</> 9.0
! or newer servers.
--- 2342,2349 ----
whether hex or traditional format is used for bytea</>
output. Libpq's PQescapeByteaConn()</> function automatically
uses the hex format when connected to PostgreSQL</> 9.0
! or newer servers. However, pre-9.0 libpq versions will not
! correctly process hex format from newer servers.
Re: [GENERAL] Gripe: bytea_output default => data corruption
От:
ljb <ljb9832@pobox.com>
Дата:
bruce@momjian.us wrote:
> Here is an updated patch, again to be backpatched to 9.0.
Getting better.
In change block 1, after PQescapeString synopsis:
Change: "does not take a Pgconn or error parameters"
to: "does not take Pgconn or error parameters"
In change block 4, PQescapeBytea, I don't like the change.
This: "is that PQescapeBytea does not take a PGconn or error parameters"
is wrong. PQescapeByteaConn does not take an 'error' parameter either.
Also "no way to report error conditions" is wrong. It returns NULL on an
error; it just cannot return a specific error message. (Which is OK because
there is only one, unlikely error: out of memory.)
Reading the before/after text here, I don't see anything that needs
changing except for dropping the reference to single-threaded (same as for
PQescapeString). I suggest going back to the original text and just
changing:
From: "can be used safely in single-threaded client programs that work with
only one"
to: "can only be used safely in client programs that use a single"
(Sorry for the trouble. I should have fully commented on the first one, but
I find it hard to see the changes with these "git" diffs of SGML.)
Re: [GENERAL] Gripe: bytea_output default => data corruption
От:
Bruce Momjian <bruce@momjian.us>
Дата:
ljb wrote:
> bruce@momjian.us wrote:
> > Here is an updated patch, again to be backpatched to 9.0.
>
> Getting better.
>
> In change block 1, after PQescapeString synopsis:
> Change: "does not take a Pgconn or error parameters"
> to: "does not take Pgconn or error parameters"
>
> In change block 4, PQescapeBytea, I don't like the change.
> This: "is that PQescapeBytea does not take a PGconn or error parameters"
> is wrong. PQescapeByteaConn does not take an 'error' parameter either.
> Also "no way to report error conditions" is wrong. It returns NULL on an
> error; it just cannot return a specific error message. (Which is OK because
> there is only one, unlikely error: out of memory.)
>
> Reading the before/after text here, I don't see anything that needs
> changing except for dropping the reference to single-threaded (same as for
> PQescapeString). I suggest going back to the original text and just
> changing:
> From: "can be used safely in single-threaded client programs that work with
> only one"
> to: "can only be used safely in client programs that use a single"
>
> (Sorry for the trouble. I should have fully commented on the first one, but
> I find it hard to see the changes with these "git" diffs of SGML.)
OK, updated version attached. I keep most of the new patch for the
PQescapeBytea docs rather than use most of the original so it would
match text for other escape functions.
--
Bruce Momjian http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index e78d708..b2ab555 100644
*** a/doc/src/sgml/libpq.sgml
--- b/doc/src/sgml/libpq.sgml
*************** size_t PQescapeStringConn(PGconn *conn,
*** 3371,3385 ****
size_t PQescapeString (char *to, const char *from, size_t length);
! PQescapeString</> is an older, deprecated version of
! PQescapeStringConn</>; the difference is that it does
! not take conn</> or error</> parameters.
Because of this, it cannot adjust its behavior depending on the
connection properties (such as character encoding) and therefore
it might give the wrong results</>. Also, it has no way
--- 3371,3387 ----
+ PQescapeString</> is an older, deprecated version of
+ PQescapeStringConn</>.
size_t PQescapeString (char *to, const char *from, size_t length);
! The only difference from PQescapeStringConn</> is that
! PQescapeString</> does not take PGconn</>
! or error</> parameters.
Because of this, it cannot adjust its behavior depending on the
connection properties (such as character encoding) and therefore
it might give the wrong results</>. Also, it has no way
*************** size_t PQescapeString (char *to, const c
*** 3387,3393 ****
! PQescapeString</> can be used safely in single-threaded
client programs that work with only one PostgreSQL</>
connection at a time (in this case it can find out what it needs to
know behind the scenes</>). In other contexts it is a security
--- 3389,3395 ----
! PQescapeString</> can be used safely in
client programs that work with only one PostgreSQL</>
connection at a time (in this case it can find out what it needs to
know behind the scenes</>). In other contexts it is a security
*************** unsigned char *PQescapeByteaConn(PGconn
*** 3419,3434 ****
! Certain byte values must be escaped (but all
! byte values can be escaped) when used as part
! of a bytea literal in an SQL
! statement. In general, to escape a byte, it is converted into the
! three digit octal number equal to the octet value, and preceded by
! usually two backslashes. The single quote ('</>) and backslash
! (\</>) characters have special alternative escape
! sequences. See for more
! information. PQescapeByteaConn performs this
! operation, escaping only the minimally required bytes.
--- 3421,3431 ----
! Certain byte values must be escaped when used as part of a
! bytea literal in an SQL statement.
! PQescapeByteaConn escapes bytes using
! either hex encoding or backslash escaping. See for more information.
*************** unsigned char *PQescapeBytea(const unsig
*** 3485,3504 ****
The only difference from PQescapeByteaConn</> is that
PQescapeBytea</> does not take a PGconn</>
! parameter. Because of this, it cannot adjust its behavior
! depending on the connection properties (in particular, whether
! standard-conforming strings are enabled) and therefore
! it might give the wrong results</>. Also, it has no
! way to return an error message on failure.
!
!
!
! PQescapeBytea</> can be used safely in single-threaded
! client programs that work with only one PostgreSQL</>
! connection at a time (in this case it can find out what it needs
! to know behind the scenes</>). In other contexts it is
! a security hazard and should be avoided in favor of
! PQescapeByteaConn</>.
--- 3482,3494 ----
The only difference from PQescapeByteaConn</> is that
PQescapeBytea</> does not take a PGconn</>
! parameter. Because of this, PQescapeBytea</> can
! only be used safely in client programs that use a single
! PostgreSQL</> connection at a time (in this case
! it can find out what it needs to know behind the
! scenes</>). It might give the wrong results</> if
! used in programs that use multiple database connections (use
! PQescapeByteaConn</> in such cases).
diff --git a/doc/src/sgml/release-9.0.sgml b/doc/src/sgml/release-9.0.sgml
index 2288f1b..c43f142 100644
*** a/doc/src/sgml/release-9.0.sgml
--- b/doc/src/sgml/release-9.0.sgml
***************
*** 2342,2348 ****
whether hex or traditional format is used for bytea</>
output. Libpq's PQescapeByteaConn()</> function automatically
uses the hex format when connected to PostgreSQL</> 9.0
! or newer servers.
--- 2342,2349 ----
whether hex or traditional format is used for bytea</>
output. Libpq's PQescapeByteaConn()</> function automatically
uses the hex format when connected to PostgreSQL</> 9.0
! or newer servers. However, pre-9.0 libpq versions will not
! correctly process hex format from newer servers.
Re: [GENERAL] Gripe: bytea_output default => data corruption
От:
ljb <ljb9832@pobox.com>
Дата:
bruce@momjian.us wrote: >... > OK, updated version attached. I keep most of the new patch for the > PQescapeBytea docs rather than use most of the original so it would > match text for other escape functions. Looks good to me.
Re: [GENERAL] Gripe: bytea_output default => data corruption
От:
Bruce Momjian <bruce@momjian.us>
Дата:
ljb wrote: > bruce@momjian.us wrote: > >... > > OK, updated version attached. I keep most of the new patch for the > > PQescapeBytea docs rather than use most of the original so it would > > match text for other escape functions. > > Looks good to me. Thanks for the reviews. Applied and backpatched to 9.0 (once git comes back online). -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +