Обсуждение: format of pg_upgrade loadable_libraries warning
Regarding the previous thread and commit here: https://www.postgresql.org/message-id/flat/20180713162815.GA3835%40momjian.us https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=60e3bd1d7f92430b24b710ecf0559656eb8ed499 I'm suggesting to reformat the warning, which I found to be misleading: |could not load library "$libdir/pgfincore": ERROR: could not access file "$libdir/pgfincore": No such file or directory |Database: postgres |Database: too To me that reads as "error message" followed by successful processing of two, named database, and not "error message followed by list of databases for which that error was experienced". Essentially, the database names are themselves the "error", and the message is a prefix indicating the library version; but normally, error-looking things are output without a "prefix", since they weren't anticipated. The existing check is optimized to check each library once, but then outputs each database which would try to load it. That's an implementation detail, but adds to confusion, since it shows a single error-looking thing which might apply to multiple DBs (not obvious to me that it's associated with an DB at all). That leads me to believe that after I "DROP EXTENSION" once, I can reasonably expect the upgrade to complete, which has a good chance of being wrong, and is exactly what the patch was intended to avoid :( To reproduce: $ /usr/pgsql-11/bin/initdb -D ./pgtestlib11 $ /usr/pgsql-12/bin/initdb -D ./pgtestlib12 $ /usr/pgsql-11/bin/pg_ctl -D ./pgtestlib11 -o '-c port=5678 -c unix_socket_directories=/tmp' start $ psql postgres -h /tmp -p 5678 -c 'CREATE EXTENSION pgfincore' -c 'CREATE DATABASE too' $ psql too -h /tmp -p 5678 -c 'CREATE EXTENSION pgfincore' $ /usr/pgsql-11/bin/pg_ctl -D ./pgtestlib11 stop $ /usr/pgsql-12/bin/pg_upgrade -b /usr/pgsql-11/bin -B /usr/pgsql-12/bin -d ./pgtestlib11 -D pgtestlib12 $ cat loadable_libraries.txt Could not load library "$libdir/pgfincore": ERROR: could not access file "$libdir/pgfincore": No such file or directory Database: postgres Database: too I concede that the situation is clearer if there are multiple libraries causing errors, especially in overlapping list of databases: |[pryzbyj@database ~]$ cat loadable_libraries.txt |could not load library "$libdir/pg_repack": ERROR: could not access file "$libdir/pg_repack": No such file or directory |Database: postgres |Database: too |could not load library "$libdir/pgfincore": ERROR: could not access file "$libdir/pgfincore": No such file or directory |Database: postgres |Database: too I think the list of databases should be formatted to indicate its association with the preceding error by indentation and verbage, or larger refactoring to present in a list, like: "Databases with library which failed to load: %s: %s", PQerrorMessage(conn), list_of_dbs_loading_that_lib Justin
Вложения
On Wed, Oct 2, 2019 at 12:23:37PM -0500, Justin Pryzby wrote: > Regarding the previous thread and commit here: > https://www.postgresql.org/message-id/flat/20180713162815.GA3835%40momjian.us > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=60e3bd1d7f92430b24b710ecf0559656eb8ed499 > > I'm suggesting to reformat the warning, which I found to be misleading: > > |could not load library "$libdir/pgfincore": ERROR: could not access file "$libdir/pgfincore": No such file or directory > |Database: postgres > |Database: too > > To me that reads as "error message" followed by successful processing of two, > named database, and not "error message followed by list of databases for which > that error was experienced". Essentially, the database names are themselves > the "error", and the message is a prefix indicating the library version; but > normally, error-looking things are output without a "prefix", since they > weren't anticipated. > > The existing check is optimized to check each library once, but then outputs > each database which would try to load it. That's an implementation detail, but > adds to confusion, since it shows a single error-looking thing which might > apply to multiple DBs (not obvious to me that it's associated with an DB at > all). That leads me to believe that after I "DROP EXTENSION" once, I can > reasonably expect the upgrade to complete, which has a good chance of being > wrong, and is exactly what the patch was intended to avoid :( Understood. This is a general problem with the way pg_upgrade displays errors and the databases/objects associated with them. The attached patch fixes the output text to say "in database", e.g.: Could not load library "$libdir/pgfincore": ERROR: could not access file "$libdir/pgfincore": No such file or directory in database: postgres in database: too Would intenting help too? I am inclined to fix this only head, and not to backpatch the change. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
Вложения
On Fri, Oct 04, 2019 at 05:37:46PM -0400, Bruce Momjian wrote: > On Wed, Oct 2, 2019 at 12:23:37PM -0500, Justin Pryzby wrote: > > Regarding the previous thread and commit here: > > https://www.postgresql.org/message-id/flat/20180713162815.GA3835%40momjian.us > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=60e3bd1d7f92430b24b710ecf0559656eb8ed499 > > > > I'm suggesting to reformat the warning, which I found to be misleading: > > Understood. This is a general problem with the way pg_upgrade displays > errors and the databases/objects associated with them. The attached > patch fixes the output text to say "in database", e.g.: > > Could not load library "$libdir/pgfincore": ERROR: could not access file "$libdir/pgfincore": No such file or directory > in database: postgres > in database: too > > Would intenting help too? I am inclined to fix this only head, and not > to backpatch the change. Yes, indenting would also help. I would argue to include in 12.1, since 12 is what most everyone will use for upgrades, and patch for .1 will help people upgrading for 11 of the next 12 months. (But, your patch is more general than mine). Thanks, Justin
On Fri, Oct 4, 2019 at 05:40:08PM -0500, Justin Pryzby wrote: > On Fri, Oct 04, 2019 at 05:37:46PM -0400, Bruce Momjian wrote: > > On Wed, Oct 2, 2019 at 12:23:37PM -0500, Justin Pryzby wrote: > > > Regarding the previous thread and commit here: > > > https://www.postgresql.org/message-id/flat/20180713162815.GA3835%40momjian.us > > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=60e3bd1d7f92430b24b710ecf0559656eb8ed499 > > > > > > I'm suggesting to reformat the warning, which I found to be misleading: > > > > Understood. This is a general problem with the way pg_upgrade displays > > errors and the databases/objects associated with them. The attached > > patch fixes the output text to say "in database", e.g.: > > > > Could not load library "$libdir/pgfincore": ERROR: could not access file "$libdir/pgfincore": No such file or directory > > in database: postgres > > in database: too > > > > Would intenting help too? I am inclined to fix this only head, and not > > to backpatch the change. > > Yes, indenting would also help. > > I would argue to include in 12.1, since 12 is what most everyone will use for > upgrades, and patch for .1 will help people upgrading for 11 of the next 12 > months. (But, your patch is more general than mine). No, there might be tools that depend on the existing format, and this is the first report of confusion I have read. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
Bruce Momjian <bruce@momjian.us> writes:
> On Fri, Oct 4, 2019 at 05:40:08PM -0500, Justin Pryzby wrote:
>> I would argue to include in 12.1, since 12 is what most everyone will use for
>> upgrades, and patch for .1 will help people upgrading for 11 of the next 12
>> months. (But, your patch is more general than mine).
> No, there might be tools that depend on the existing format, and this is
> the first report of confusion I have read.
Translations will also lag behind any such change. Speaking of which,
it might be a good idea to include some translator: annotations to
help translators understand what context these fragmentary phrases
are used in. I'd actually say that my biggest concern with these
messages is whether they can translate into something sane.
regards, tom lane
On Fri, Oct 4, 2019 at 11:55:21PM -0400, Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > On Fri, Oct 4, 2019 at 05:40:08PM -0500, Justin Pryzby wrote:
> >> I would argue to include in 12.1, since 12 is what most everyone will use for
> >> upgrades, and patch for .1 will help people upgrading for 11 of the next 12
> >> months. (But, your patch is more general than mine).
>
> > No, there might be tools that depend on the existing format, and this is
> > the first report of confusion I have read.
>
> Translations will also lag behind any such change. Speaking of which,
> it might be a good idea to include some translator: annotations to
> help translators understand what context these fragmentary phrases
> are used in. I'd actually say that my biggest concern with these
> messages is whether they can translate into something sane.
Uh, I looked at the pg_ugprade code and the error message is:
pg_fatal("Your installation contains \"contrib/isn\" functions which rely on the\n"
"bigint data type. Your old and new clusters pass bigint values\n"
"differently so this cluster cannot currently be upgraded. You can\n"
"manually upgrade databases that use \"contrib/isn\" facilities and remove\n"
"\"contrib/isn\" from the old cluster and restart the upgrade. A list of\n"
"the problem functions is in the file:\n"
" %s\n\n", output_path);
and the "in database" (which I have changed to capitalized "In database"
in the attached patch), looks like:
fprintf(script, "In database: %s\n", active_db->db_name);
meaning it _isn't_ an output error message, but rather something that
appears in an error file. I don't think either of these are translated.
Is that wrong?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
Вложения
On Mon, Oct 7, 2019 at 01:47:57PM -0400, Bruce Momjian wrote:
> On Fri, Oct 4, 2019 at 11:55:21PM -0400, Tom Lane wrote:
> > Bruce Momjian <bruce@momjian.us> writes:
> > > On Fri, Oct 4, 2019 at 05:40:08PM -0500, Justin Pryzby wrote:
> > >> I would argue to include in 12.1, since 12 is what most everyone will use for
> > >> upgrades, and patch for .1 will help people upgrading for 11 of the next 12
> > >> months. (But, your patch is more general than mine).
> >
> > > No, there might be tools that depend on the existing format, and this is
> > > the first report of confusion I have read.
> >
> > Translations will also lag behind any such change. Speaking of which,
> > it might be a good idea to include some translator: annotations to
> > help translators understand what context these fragmentary phrases
> > are used in. I'd actually say that my biggest concern with these
> > messages is whether they can translate into something sane.
>
> Uh, I looked at the pg_ugprade code and the error message is:
>
> pg_fatal("Your installation contains \"contrib/isn\" functions which rely on the\n"
> "bigint data type. Your old and new clusters pass bigint values\n"
> "differently so this cluster cannot currently be upgraded. You can\n"
> "manually upgrade databases that use \"contrib/isn\" facilities and remove\n"
> "\"contrib/isn\" from the old cluster and restart the upgrade. A list of\n"
> "the problem functions is in the file:\n"
> " %s\n\n", output_path);
>
> and the "in database" (which I have changed to capitalized "In database"
> in the attached patch), looks like:
>
> fprintf(script, "In database: %s\n", active_db->db_name);
>
> meaning it _isn't_ an output error message, but rather something that
> appears in an error file. I don't think either of these are translated.
> Is that wrong?
Patch applied to head.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
On 2019-Oct-07, Bruce Momjian wrote:
> Uh, I looked at the pg_ugprade code and the error message is:
>
> pg_fatal("Your installation contains \"contrib/isn\" functions which rely on the\n"
> "bigint data type. Your old and new clusters pass bigint values\n"
> "differently so this cluster cannot currently be upgraded. You can\n"
> "manually upgrade databases that use \"contrib/isn\" facilities and remove\n"
> "\"contrib/isn\" from the old cluster and restart the upgrade. A list of\n"
> "the problem functions is in the file:\n"
> " %s\n\n", output_path);
>
> and the "in database" (which I have changed to capitalized "In database"
> in the attached patch), looks like:
>
> fprintf(script, "In database: %s\n", active_db->db_name);
>
> meaning it _isn't_ an output error message, but rather something that
> appears in an error file. I don't think either of these are translated.
> Is that wrong?
pg_fatal is a "gettext trigger" (see nls.mk), so that part of the
message is definitely translated. And the fprintf format string should
be decorated with _() in order to make translatable too; otherwise the
message is only half-translated when it appears in the pg_upgrade log,
which is not nice. This should look like:
if (!db_used)
{
/* translator: This is an error message indicator */
fprintf(script, _("In database: %s\n"), active_db->db_name);
db_used = true;
}
fprintf(script, " %s.%s\n",
BTW, how is one supposed to "manually upgrade databases that use
contrib/isb"? This part is not very clear.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> On 2019-Oct-07, Bruce Momjian wrote:
>> and the "in database" (which I have changed to capitalized "In database"
>> in the attached patch), looks like:
>> fprintf(script, "In database: %s\n", active_db->db_name);
>> meaning it _isn't_ an output error message, but rather something that
>> appears in an error file. I don't think either of these are translated.
>> Is that wrong?
> pg_fatal is a "gettext trigger" (see nls.mk), so that part of the
> message is definitely translated.
Right, but Bruce's point is that what goes into the separate output
file listing problem cases is not translated, and never has been.
Maybe we should start doing so, but that would be a distinct issue.
I'm not really sure that we should translate it, anyway --- could
there be anyone out there who is using tools to process these files?
> BTW, how is one supposed to "manually upgrade databases that use
> contrib/isb"? This part is not very clear.
Agreed, the pg_fatal message is claiming that you can do something
without really providing any concrete instructions for it. I'm not
sure that that's helpful.
regards, tom lane
On Thu, Nov 14, 2019 at 04:06:52PM -0300, Alvaro Herrera wrote: > BTW, how is one supposed to "manually upgrade databases that use > contrib/isb"? This part is not very clear. I thought you would dump out databases that use isn, drop those databases, use pg_upgrade for the remaining databases, then load the dumped database. Attached is a patch that improves the wording. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
Вложения
On Thu, Nov 14, 2019 at 02:46:29PM -0500, Tom Lane wrote: > Alvaro Herrera <alvherre@2ndquadrant.com> writes: > > On 2019-Oct-07, Bruce Momjian wrote: > >> and the "in database" (which I have changed to capitalized "In database" > >> in the attached patch), looks like: > >> fprintf(script, "In database: %s\n", active_db->db_name); > >> meaning it _isn't_ an output error message, but rather something that > >> appears in an error file. I don't think either of these are translated. > >> Is that wrong? > > > pg_fatal is a "gettext trigger" (see nls.mk), so that part of the > > message is definitely translated. > > Right, but Bruce's point is that what goes into the separate output > file listing problem cases is not translated, and never has been. > Maybe we should start doing so, but that would be a distinct issue. > I'm not really sure that we should translate it, anyway --- could > there be anyone out there who is using tools to process these files? Yes, we are lacking in all these output files so if we do one, we should do them all. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
Bruce Momjian <bruce@momjian.us> writes:
> On Thu, Nov 14, 2019 at 04:06:52PM -0300, Alvaro Herrera wrote:
>> BTW, how is one supposed to "manually upgrade databases that use
>> contrib/isb"? This part is not very clear.
> I thought you would dump out databases that use isn, drop those
> databases, use pg_upgrade for the remaining databases, then load the
> dumped database. Attached is a patch that improves the wording.
That's better wording, but do we need similar for any of the other
not-supported checks?
regards, tom lane
On Thu, Nov 14, 2019 at 05:49:12PM -0500, Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > On Thu, Nov 14, 2019 at 04:06:52PM -0300, Alvaro Herrera wrote: > >> BTW, how is one supposed to "manually upgrade databases that use > >> contrib/isb"? This part is not very clear. > > > I thought you would dump out databases that use isn, drop those > > databases, use pg_upgrade for the remaining databases, then load the > > dumped database. Attached is a patch that improves the wording. > > That's better wording, but do we need similar for any of the other > not-supported checks? I don't think so. The other checks are checking for the _use_ of certain things in user objects, and the user objects can be dropped, while this check checks for the existence of _functions_ from an extension that must be uninstalled. I assume the extension can be uninstalled in the database, but I assume something uses it and that telling people to find all the users of the extension then dropping it is too complex to describe. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
> On 14 Nov 2019, at 23:16, Bruce Momjian <bruce@momjian.us> wrote: > > On Thu, Nov 14, 2019 at 04:06:52PM -0300, Alvaro Herrera wrote: >> BTW, how is one supposed to "manually upgrade databases that use >> contrib/isb"? This part is not very clear. > > I thought you would dump out databases that use isn, drop those > databases, use pg_upgrade for the remaining databases, then load the > dumped database. Attached is a patch that improves the wording. I agree with this patch, that's much a more informative message. There is one tiny typo in the patch: s/laster/later/ + "cluster, drop them, restart the upgrade, and restore them laster. A\n" cheers ./daniel
On Fri, Nov 15, 2019 at 12:32:55AM +0100, Daniel Gustafsson wrote: > > On 14 Nov 2019, at 23:16, Bruce Momjian <bruce@momjian.us> wrote: > > > > On Thu, Nov 14, 2019 at 04:06:52PM -0300, Alvaro Herrera wrote: > >> BTW, how is one supposed to "manually upgrade databases that use > >> contrib/isb"? This part is not very clear. > > > > I thought you would dump out databases that use isn, drop those > > databases, use pg_upgrade for the remaining databases, then load the > > dumped database. Attached is a patch that improves the wording. > > I agree with this patch, that's much a more informative message. > > There is one tiny typo in the patch: s/laster/later/ > > + "cluster, drop them, restart the upgrade, and restore them laster. A\n" I have applied the patch, with improved wording. I only applied this to PG 13 since I was worried old tools might be checking for the old error text. Should this be backpatched more? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
> On 28 Nov 2019, at 02:26, Bruce Momjian <bruce@momjian.us> wrote: > I have applied the patch, with improved wording. I only applied this to > PG 13 since I was worried old tools might be checking for the old error > text. Should this be backpatched more? I don't think it's unreasonable to assume that there are lots of inhouse tooling for pg_upgrade orchestration which grep for specific messages. Stopping at 13 seems perfectly reasonable for this change. cheers ./daniel