Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

Поиск
Список
Период
Сортировка
Искать

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Greg Sabino Mullane <htamfids@gmail.com>
Дата:
My review. Overall, this looks really nice and a much needed feature. I will definitely be using this a lot in the future :)

It cleanly compiled on top of 97a18c22 (from Fri Jul 10 22:34:24 2026), and I was able to successfully do the upgrade via the new --initdb flag for versions 11 through 18. Version 10 did not work, but that is not the fault of this patch. But the output for completeness, since we still support upgrading from v10:

   Inspecting old cluster locale for new cluster creation
   invalid connection option "max_protocol_version"

I also verified that it correctly maintains the checksum setting, the locale, weird WAL segment sizes, and the new icu and builtin locale options. However, the tests do none of those checks, and they really ought to, as that's an important part of what this feature provides.

> Extra initdb options can be passed via the existing -O flag and will be
> forwarded to the initdb invocation.

I don't like this part, as there is not a 100% overlap between what postmaster accepts and what initdb accepts. The -c options are fine, but things like -O "-B 12345" cause a failure. I'd simply not add the -O options here: if people really need them, they can run pg_upgrade without the --initdb flag. Alternatively, only allow -c items, but that's a real pain to code that.

> +static void create_new_cluster_via_initdb(void);

Nice, I love descriptive function names.

> + * resolve_new_bindir()

Good idea, well executed.

> +/* Refuse to overwrite an existing cluster. */

Move this up to just after the other check that initdb is present?

> + snprintf(verfile, sizeof(verfile), "%s/PG_VERSION",

I suppose this is slightly better than just checking if the dir is empty or not, as initdb itself will later fail if non-empty but does not contain PG_VERSION. So a weak +1. Sort of has test coverage (but see below)

> + * Verify that initdb is present and executable before doing any work.

Also not tested at all (may be overkill to do so, but pointing out for completeness)

> +# Copyright (c) 2022-2025, PostgreSQL Global Development Group

s/2025/2026/  - or perhaps just say "2026" as this is a new file

> +# --initdb must refuse to clobber an already-populated data directory.
> +command_fails(

Would be nice if we confirmed this was failing due to the new PG_VERSION check, versus the initdb "ain't gonna write to non empty dir" error.

Cheers,
Greg

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Hüseyin Demir <huseyin.d3r@gmail.com>
Дата:
Hi,

> Hi,
>
> This patch adds an --initdb option to pg_upgrade that automates the initdb step currently required before running pg_upgrade.
>
> Problem
> —————
> Before running pg_upgrade, users must manually run initdb with options that exactly match the old cluster: WAL segment size, data checksum setting, encoding, and locale. Getting these right is error-prone. A mismatch results in a confusing check_control_data() failure after the user has already invested time in setting up the new cluster. A related question was raised before [1], where Jeff Davis discussed whether pg_upgrade should perform initdb itself rather than requiring a pre-initialized cluster.
>
> Solution
> ————
> With --initdb, pg_upgrade handles this automatically. It derives the WAL segment size and checksum setting from pg_control and invokes initdb with the correct flags. The option refuses to proceed if the new cluster already exists.
>

Thanks for the patch but I have a couple of concerns that will require
further discussion.

Since we include the initdb option to pg_upgrade it can cause orphan
directories that users must manually solve. Partial initdb failures
leave orphaned clusters.

Moreover, in the patch the old cluster is being started/stopped twice
meaning that the pg_upgrade needs to manage the states. This will
cause unpredictable behavior on retry and make things really hard to
debug I believe.

On the other side, the responsibility of pg_upgrade shouldn't contain
initializing a new cluster and deciding the locale,encoding, and
checksums. If the idea is accepted we should focus on documentation
and avoid the risks it will introduce during the upgrade.

Regards,
Demir.


Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Bohyun Lee <bohyun.lee@databricks.com>
Дата:
Вложения:

Thanks for the review, Greg. Glad it's useful, and thanks for trying it across the version range.


Attached is v2, which addresses your points:


- Added test coverage for the settings the feature is really about. The test now initdb's the old cluster with data checksums and a non-default WAL segment size, and after the upgrade checks that the new cluster inherited the checksum setting, WAL segment size, encoding, collation, ctype, and locale provider.


- The overwrite test now confirms the failure comes from pg_upgrade's own PG_VERSION check ("already contains a database system") rather than initdb's non-empty-directory error.


- Added a test for the "initdb not present" case (pointing --new-bindir at an empty dir), since it was easy enough to cover.


- Reworked the -O handling. Rather than dropping it entirely, I took your alternative: only "-c name=value" options are forwarded to initdb now, and anything else is skipped with a warning. So -O "-c work_mem=..." works, and -O "-B 12345" no longer breaks initdb. 

That keeps GUC pass-through working without pretending initdb accepts the full postmaster option set.


- Moved the "refuse to overwrite" check up to right after the initdb-present check.


- Fixed the copyright year on the new test file.


 On the v10 failure ("invalid connection option max_protocol_version"): that comes from stock pg_upgrade, which adds max_protocol_version=3.0 to the connection string for pre-PG11 servers (server.c). It's not introduced by --initdb, the same connection is made in the normal flow, so I've left it out of scope for this patch.


The patch is pgindent-clean and the TAP test passes. For convenience, the branch is also on GitHub:

  github.com/LeeBohyun/postgres/tree/pg_upgrade_initdb


Best regards,

Bohyun


On Fri, Jul 10, 2026 at 7:24 PM Greg Sabino Mullane <htamfids@gmail.com> wrote:
My review. Overall, this looks really nice and a much needed feature. I will definitely be using this a lot in the future :)

It cleanly compiled on top of 97a18c22 (from Fri Jul 10 22:34:24 2026), and I was able to successfully do the upgrade via the new --initdb flag for versions 11 through 18. Version 10 did not work, but that is not the fault of this patch. But the output for completeness, since we still support upgrading from v10:

   Inspecting old cluster locale for new cluster creation
   invalid connection option "max_protocol_version"

I also verified that it correctly maintains the checksum setting, the locale, weird WAL segment sizes, and the new icu and builtin locale options. However, the tests do none of those checks, and they really ought to, as that's an important part of what this feature provides.

> Extra initdb options can be passed via the existing -O flag and will be
> forwarded to the initdb invocation.

I don't like this part, as there is not a 100% overlap between what postmaster accepts and what initdb accepts. The -c options are fine, but things like -O "-B 12345" cause a failure. I'd simply not add the -O options here: if people really need them, they can run pg_upgrade without the --initdb flag. Alternatively, only allow -c items, but that's a real pain to code that.

> +static void create_new_cluster_via_initdb(void);

Nice, I love descriptive function names.

> + * resolve_new_bindir()

Good idea, well executed.

> +/* Refuse to overwrite an existing cluster. */

Move this up to just after the other check that initdb is present?

> + snprintf(verfile, sizeof(verfile), "%s/PG_VERSION",

I suppose this is slightly better than just checking if the dir is empty or not, as initdb itself will later fail if non-empty but does not contain PG_VERSION. So a weak +1. Sort of has test coverage (but see below)

> + * Verify that initdb is present and executable before doing any work.

Also not tested at all (may be overkill to do so, but pointing out for completeness)

> +# Copyright (c) 2022-2025, PostgreSQL Global Development Group

s/2025/2026/  - or perhaps just say "2026" as this is a new file

> +# --initdb must refuse to clobber an already-populated data directory.
> +command_fails(

Would be nice if we confirmed this was failing due to the new PG_VERSION check, versus the initdb "ain't gonna write to non empty dir" error.

Cheers,
Greg

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Daniel Gustafsson <daniel@yesql.se>
Дата:
I am not convinced that all this complexity and overhead isn't better suited
for external wrappers to pg_upgrade like Debian's pg_upgradecluster etc.

+	if (user_opts.initdb_new_cluster)
+		create_new_cluster_via_initdb();
+
 	adjust_data_dir(&new_cluster);

If pg_upgrade controls how initdb was invoked for the new cluster, shouldn't it
work such that adjust_data_dir isn't required?


+	/*
+	 * get_control_data() selects pg_resetwal vs. pg_resetxlog via
+	 * bin_version, which check_bindir() normally fills in later.  Seed it now
+	 * so the right binary name is used in this early call.
+	 */
+	if (old_cluster.bin_version == 0)
+		old_cluster.bin_version = old_cluster.major_version;
+
+	get_control_data(&old_cluster);

This can't be done unconditionally, the old cluster can still be running at
this point.  For example if someone wants to do a live-check:

    $ ./bin/pg_upgrade -b ./bin/ -B ./bin/ -d ./data_old/ -D ./data_new/ --check --initdb

    The source cluster was not shut down cleanly, state reported as: "in production"
    Failure, exiting

get_control_data is big, expensive, and really designed to be run once.  I
don't think it's Ok to run it an extra time here without at least being able to
tell the later invocation that it has already been executed.  Also,
check_bindir() as referred to in the comment does not exist.


+ prep_status("Inspecting old cluster locale for new cluster creation");
+ start_postmaster(&old_cluster, true);
+ get_template0_info(&old_cluster);
+ stop_postmaster(false);
+ check_ok();

Again, cannot be done unconditionally.


+	 * Users needing options that only the postmaster accepts can create the
+	 * new cluster manually and omit --initdb.

This should probably be expanded upon in the documentation.

--
Daniel Gustafsson



Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Bohyun Lee <bohyun.lee@databricks.com>
Дата:
Вложения:
Hi Daniel,

Thanks for the review. v3 attached.

On external wrapper vs. in-tree: the point of --initdb is deriving the new cluster's settings (WAL segment size, checksums, encoding, locale) from the old cluster, exactly what people get wrong by hand and only discover when check_control_data() fails. pg_upgrade already reads the old control data, so doing it here reuses that rather than making a wrapper rediscover it. I'm open to being convinced otherwise.

The rest:
- --check --initdb: you're right it was broken (ran before live-check was determined). Since --check is read-only and --initdb creates the cluster, the combination doesn't make sense — v3 rejects it at option parsing, which removes the live-check problem entirely.
- get_control_data() twice: the early --initdb read populates the old cluster's control data, so the later call in check_cluster_compatibility() returns early when ctrl_ver is already set. New cluster is still read fresh.
- adjust_data_dir(): moved adjust_data_dir(&new_cluster) before the initdb call so a config-only -D is resolved first.
- check_bindir(): meant get_bin_version(). Fixed.
- Docs: noted that only initdb-accepted settings are configured, postmaster-only options need manual creation, and the --check restriction.

TAP test covers the --check rejection. pgindent-clean, tests pass.
https://github.com/LeeBohyun/postgres/tree/pg_upgrade_initdb
Best regards,
Bohyun

On Tue, Jul 14, 2026 at 12:06 PM Daniel Gustafsson <daniel@yesql.se> wrote:
I am not convinced that all this complexity and overhead isn't better suited
for external wrappers to pg_upgrade like Debian's pg_upgradecluster etc.

+       if (user_opts.initdb_new_cluster)
+               create_new_cluster_via_initdb();
+
        adjust_data_dir(&new_cluster);

If pg_upgrade controls how initdb was invoked for the new cluster, shouldn't it
work such that adjust_data_dir isn't required?


+       /*
+        * get_control_data() selects pg_resetwal vs. pg_resetxlog via
+        * bin_version, which check_bindir() normally fills in later.  Seed it now
+        * so the right binary name is used in this early call.
+        */
+       if (old_cluster.bin_version == 0)
+               old_cluster.bin_version = old_cluster.major_version;
+
+       get_control_data(&old_cluster);

This can't be done unconditionally, the old cluster can still be running at
this point.  For example if someone wants to do a live-check:

    $ ./bin/pg_upgrade -b ./bin/ -B ./bin/ -d ./data_old/ -D ./data_new/ --check --initdb

    The source cluster was not shut down cleanly, state reported as: "in production"
    Failure, exiting

get_control_data is big, expensive, and really designed to be run once.  I
don't think it's Ok to run it an extra time here without at least being able to
tell the later invocation that it has already been executed.  Also,
check_bindir() as referred to in the comment does not exist.


+ prep_status("Inspecting old cluster locale for new cluster creation");
+ start_postmaster(&old_cluster, true);
+ get_template0_info(&old_cluster);
+ stop_postmaster(false);
+ check_ok();

Again, cannot be done unconditionally.


+        * Users needing options that only the postmaster accepts can create the
+        * new cluster manually and omit --initdb.

This should probably be expanded upon in the documentation.

--
Daniel Gustafsson

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Greg Sabino Mullane <htamfids@gmail.com>
Дата:
Second review pass.

Cleanly compiles and all tests pass when applied to 637aa273e (Thu Jul 16 11:56:01 2026 -0400)

/*
 * resolve_new_bindir()
 *
 * Idempotent helper: if new_cluster.bindir has not been set by the user via
 * -B, derive it from the path of the currently executing pg_upgrade binary.
 * Called early by create_new_cluster_via_initdb() so that the initdb path
 * is available before verify_directories() runs.
 */

Do we really need the last two lines? This is also called during the "normal" path, not just via create_new_cluster_via_initdb, so not sure why we are pointing it out here.


> prep_status("Inspecting old cluster locale for new cluster creation");

We are doing a lot more than locale though. How about:

prep_status("Examining old cluster settings");


> # Use non-default settings that --initdb must carry over to the new cluster
> # (derived from the old cluster's pg_control): data checksums and a non-default
> # WAL segment size.  We check below that the new cluster inherits them.
> my $oldnode = PostgreSQL::Test::Cluster->new('old_node');
> $oldnode->init(extra => [ '--data-checksums', '--wal-segsize' => '2' ]);

But checkums ARE the default now, so for a non-default setting, we should do --no-data-checksums. Would be nice to check locale as well. That gets tricky, but maybe --locale=C is nicely portable and pretty non-default?


> my $conf = $newnode->data_dir . '/postgresql.conf';
> open(my $fh, '>>', $conf) or die "could not open $conf: $!";

Easier to use $newnode->append_conf


> print $fh "listen_addresses = ''\n";
> print $fh "unix_socket_directories = '" . $newnode->host . "'\n";

What about Win32? See .e.g $use_tcp in Cluster.pm

> * Forward only "-c name=value" options from -O to initdb.  initdb accepts

I still think we just bail on supporting -O. The new code is an improvement, but fails if there are legitimate spaces in any of the args:

$ pg_upgrade --initdb -O "-c transaction_isolation='repeatable read'"

Creating new cluster with initdb                              ignoring non-"-c" option(s) passed via -O; only "-c" settings are forwarded to initdb for --initdb
sh: 1: Syntax error: Unterminated quoted string

*failure*


Cheers,
Greg

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Bohyun Lee <bohyun.lee@databricks.com>
Дата:
Вложения:

Thanks both for the reviews. v4 attached, addressing all of Greg's second-pass points and Daniel's documentation point.

Greg:

  • resolve_new_bindir(): dropped the last two comment lines.

  • prep_status text is now "Examining old cluster settings".

  • Test: since checksums are now the default, the old cluster is created with --no-data-checksums (a genuinely non-default value) and --locale=C. The post-upgrade checks confirm the new cluster inherited both, along with the WAL segment size, encoding, collation, ctype, and locale provider.

  • Test: replaced open()/print() with append_conf, and the port/socket settings now follow the framework's TCP-vs-Unix logic ($use_tcp), so Windows works too.

  • -O: dropped support with --initdb entirely, as you originally suggested. The partial "-c only" forwarding still broke on quoted values with spaces. pg_upgrade now rejects -O + --initdb during option parsing, with a test and a doc note.


Daniel:

  • Reworked the manual-initdb docs: the vague "use compatible initdb flags" is now the specific settings that must match (WAL segment size, checksums, encoding, locale), with a pointer to pg_controldata.

On in-tree vs. wrapper: these settings have to come from the old cluster's control data, which pg_upgrade already reads, a wrapper would have to rediscover the same information, and getting it wrong is exactly the failure --initdb is meant to prevent.


The patch is pgindent-clean and the full pg_upgrade TAP suite passes (including the new t/009_initdb_option.pl). 

Registered in PG20-2. Branch on GitHub: https://github.com/LeeBohyun/postgres/tree/pg_upgrade_initdb


Best regards, 

Bohyun


On Thu, Jul 16, 2026 at 8:17 PM Greg Sabino Mullane <htamfids@gmail.com> wrote:
Second review pass.

Cleanly compiles and all tests pass when applied to 637aa273e (Thu Jul 16 11:56:01 2026 -0400)

/*
 * resolve_new_bindir()
 *
 * Idempotent helper: if new_cluster.bindir has not been set by the user via
 * -B, derive it from the path of the currently executing pg_upgrade binary.
 * Called early by create_new_cluster_via_initdb() so that the initdb path
 * is available before verify_directories() runs.
 */

Do we really need the last two lines? This is also called during the "normal" path, not just via create_new_cluster_via_initdb, so not sure why we are pointing it out here.


> prep_status("Inspecting old cluster locale for new cluster creation");

We are doing a lot more than locale though. How about:

prep_status("Examining old cluster settings");


> # Use non-default settings that --initdb must carry over to the new cluster
> # (derived from the old cluster's pg_control): data checksums and a non-default
> # WAL segment size.  We check below that the new cluster inherits them.
> my $oldnode = PostgreSQL::Test::Cluster->new('old_node');
> $oldnode->init(extra => [ '--data-checksums', '--wal-segsize' => '2' ]);

But checkums ARE the default now, so for a non-default setting, we should do --no-data-checksums. Would be nice to check locale as well. That gets tricky, but maybe --locale=C is nicely portable and pretty non-default?


> my $conf = $newnode->data_dir . '/postgresql.conf';
> open(my $fh, '>>', $conf) or die "could not open $conf: $!";

Easier to use $newnode->append_conf


> print $fh "listen_addresses = ''\n";
> print $fh "unix_socket_directories = '" . $newnode->host . "'\n";

What about Win32? See .e.g $use_tcp in Cluster.pm

> * Forward only "-c name=value" options from -O to initdb.  initdb accepts

I still think we just bail on supporting -O. The new code is an improvement, but fails if there are legitimate spaces in any of the args:

$ pg_upgrade --initdb -O "-c transaction_isolation='repeatable read'"

Creating new cluster with initdb                              ignoring non-"-c" option(s) passed via -O; only "-c" settings are forwarded to initdb for --initdb
sh: 1: Syntax error: Unterminated quoted string

*failure*


Cheers,
Greg

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Hüseyin Demir <huseyin.d3r@gmail.com>
Дата:
Hi,

A few feedback about introducing the initdb option to pg_upgrade.

When I run the pg_upgrade command with the following options I'd
expect it to not create a directory and initdb.

```
pg_upgrade --check --initdb -d /old -D /new
```

So if the check is enabled it should print out the potential initdb
command/arguments/output and run the check. Because it can lead to
misunderstanding while executing the actual upgrade commands. And
operator/script has to remove the initdb directory before actual
upgrade.

When it comes to documentation we also need to add that the initdb
option starts the old server briefly to discover settings from
template0, then stops it before creating the new cluster. Moreover,
also mention that -B flag points outs to target version correctly
since there is a risk of doing an initdb with the wrong version. If -B
points to an older version, pg_upgrade may create the wrong cluster.

And the --initdb behaviour will introduce additional risks during the
upgrade. During the initdb the settings discovery connections are
allowed on database level and we don't have a guarantee that
checkpoint and vacuums won't start. This needs to be documented
clearly or we should consider eliminating them.

Regards,
Demir.

Bohyun Lee , 17 Tem 2026 Cum, 14:10
tarihinde şunu yazdı:
>
> Thanks both for the reviews. v4 attached, addressing all of Greg's second-pass points and Daniel's documentation point.
>
> Greg:
>
> resolve_new_bindir(): dropped the last two comment lines.
>
> prep_status text is now "Examining old cluster settings".
>
> Test: since checksums are now the default, the old cluster is created with --no-data-checksums (a genuinely non-default value) and --locale=C. The post-upgrade checks confirm the new cluster inherited both, along with the WAL segment size, encoding, collation, ctype, and locale provider.
>
> Test: replaced open()/print() with append_conf, and the port/socket settings now follow the framework's TCP-vs-Unix logic ($use_tcp), so Windows works too.
>
> -O: dropped support with --initdb entirely, as you originally suggested. The partial "-c only" forwarding still broke on quoted values with spaces. pg_upgrade now rejects -O + --initdb during option parsing, with a test and a doc note.
>
>
> Daniel:
>
> Reworked the manual-initdb docs: the vague "use compatible initdb flags" is now the specific settings that must match (WAL segment size, checksums, encoding, locale), with a pointer to pg_controldata.
>
> On in-tree vs. wrapper: these settings have to come from the old cluster's control data, which pg_upgrade already reads, a wrapper would have to rediscover the same information, and getting it wrong is exactly the failure --initdb is meant to prevent.
>
>
> The patch is pgindent-clean and the full pg_upgrade TAP suite passes (including the new t/009_initdb_option.pl).
>
> Registered in PG20-2. Branch on GitHub: https://github.com/LeeBohyun/postgres/tree/pg_upgrade_initdb
>
>
> Best regards,
>
> Bohyun
>
>
> On Thu, Jul 16, 2026 at 8:17 PM Greg Sabino Mullane  wrote:
>>
>> Second review pass.
>>
>> Cleanly compiles and all tests pass when applied to 637aa273e (Thu Jul 16 11:56:01 2026 -0400)
>>
>> /*
>>  * resolve_new_bindir()
>>  *
>>  * Idempotent helper: if new_cluster.bindir has not been set by the user via
>>  * -B, derive it from the path of the currently executing pg_upgrade binary.
>>  * Called early by create_new_cluster_via_initdb() so that the initdb path
>>  * is available before verify_directories() runs.
>>  */
>>
>> Do we really need the last two lines? This is also called during the "normal" path, not just via create_new_cluster_via_initdb, so not sure why we are pointing it out here.
>>
>>
>> > prep_status("Inspecting old cluster locale for new cluster creation");
>>
>> We are doing a lot more than locale though. How about:
>>
>> prep_status("Examining old cluster settings");
>>
>>
>> > # Use non-default settings that --initdb must carry over to the new cluster
>> > # (derived from the old cluster's pg_control): data checksums and a non-default
>> > # WAL segment size.  We check below that the new cluster inherits them.
>> > my $oldnode = PostgreSQL::Test::Cluster->new('old_node');
>> > $oldnode->init(extra => [ '--data-checksums', '--wal-segsize' => '2' ]);
>>
>> But checkums ARE the default now, so for a non-default setting, we should do --no-data-checksums. Would be nice to check locale as well. That gets tricky, but maybe --locale=C is nicely portable and pretty non-default?
>>
>>
>> > my $conf = $newnode->data_dir . '/postgresql.conf';
>> > open(my $fh, '>>', $conf) or die "could not open $conf: $!";
>>
>> Easier to use $newnode->append_conf
>>
>>
>> > print $fh "listen_addresses = ''\n";
>> > print $fh "unix_socket_directories = '" . $newnode->host . "'\n";
>>
>> What about Win32? See .e.g $use_tcp in Cluster.pm
>>
>> > * Forward only "-c name=value" options from -O to initdb.  initdb accepts
>>
>> I still think we just bail on supporting -O. The new code is an improvement, but fails if there are legitimate spaces in any of the args:
>>
>> $ pg_upgrade --initdb -O "-c transaction_isolation='repeatable read'"
>>
>> Creating new cluster with initdb                              ignoring non-"-c" option(s) passed via -O; only "-c" settings are forwarded to initdb for --initdb
>> sh: 1: Syntax error: Unterminated quoted string
>>
>> *failure*
>>
>>
>> Cheers,
>> Greg


Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Daniel Gustafsson <daniel@yesql.se>
Дата:
> On 17 Jul 2026, at 14:10, Bohyun Lee  wrote:

>     • Reworked the manual-initdb docs: the vague "use compatible initdb flags" is now the specific settings that must match (WAL segment size, checksums, encoding, locale), with a pointer to pg_controldata.

Regardless of the rest of the patch, I think we should apply something like
this to the docs.  I propose you extract this portion into a separate 0001 in
this patchset so that it can be considered individually.

I haven't had a chance to look at the new version more than at skim-level but
the below caught my eye.

+	initPQExpBuffer(&cmd);
+	appendPQExpBuffer(&cmd, "\"%s/initdb\" -D \"%s\" -N",
+					  new_cluster.bindir, new_cluster.pgdata);
+	appendPQExpBuffer(&cmd, " -U \"%s\"", os_info.user);

This will inject the username specified in the (untrusted input) commandline
into the initdb command and execute it.  While that will work in benign cases,
it seems like a Booby Tables pattern which is best left in the coding practices
of the past and in XKCD?

--
Daniel Gustafsson



Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Hüseyin Demir <huseyin.d3r@gmail.com>
Дата:
Вложения:
Hi,

> v5 resolved this as a dry run rather than a hard block.  --check
> --initdb reports the initdb command it would run and validates the
> preconditions (correct new-binary version, empty target directory, old
> cluster reachable) without creating the new cluster. v6 keeps that,
> and the docs now describe it.

Thanks, appreciated.

> > I'm not clear what pointing this out for --initdb only accomplishes.
> > A wrong -B is going to cause problems regardless of the mode.
> I agree, and I believe this meets Demir's point about orphan directories.
> v5 checks the new-binary version up front, so a wrong -B fails before anything is
> written. An atexit handler removes the created directory if a later
> check fails, up until the point where real data has been copied. So
> there is no need for a -B note specific to --initdb in the docs.
> v6 carries both.

Got it, great.

>
> > Eliminating what exactly? And why would a vacuum or checkpoint cause
> > problems?
> The old server is started here with the same start_postmaster() that
> pg_upgrade uses elsewhere -- including -b, which disables autovacuum --
> only to read template0's encoding and locale, then stopped before the
> new cluster is created.  So autovacuum will not run, and a checkpoint
> would be harmless.  It is a brief, read-only start using pg_upgrade's
> existing mechanism, so it adds no risk beyond the old-server starts
> pg_upgrade already performs.  The docs now mention this brief start.

Thanks for handling this.

> Beyond the review points, v6 also tightens two things I noticed while
> reworking this:

> - The empty-directory guard now rejects any non-empty target directory,
>   not just one already containing a PG_VERSION file. This closes a gap
>   in the orphan-cleanup path. The cleanup handler removes the whole new
>   data directory on failure, so it must not run against a directory that
>   already held the user's files.

> - Because --initdb reads template0 by briefly starting the old server, it
>   now checks in advance that the old server is shut down and fails with the
>   usual "shut down that postmaster" message, rather than failing later
>   when pg_resetwal refuses to run against a live server.

Good catches, I have no objections to v6.

Regards,
Demir.


pgsql: Fix -Wshadow=local warnings

От:
Peter Eisentraut <peter@eisentraut.org>
Дата:
Вложения:
Fix -Wshadow=local warnings

This fixes issues with the same variable name being used in the same
function for two different variables.  (In some cases, one of the uses
is a function argument.)  The fix is in most cases to rename one or
both of them.  The individual choice depends on conventions in the
surrounding code.  Often, making both variable names more specific is
the best choice.  In a few cases, the fix is to move one of the
variables to a lower scope, so that it no longer conflicts with the
other.

Since these are all local-variable conflicts, the changes in different
files in this patch are independent of each other.

These issues correspond to the gcc warning option -Wshadow=local,
which is not currently used but could be activated after this.

Reviewed-by: Peter Geoghegan 
Reviewed-by: Chao Li 
Discussion: https://www.postgresql.org/message-id/flat/84b6f128-91f5-480e-8a9e-7d0e8f538cea%40eisentraut.org

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/f9fc9a78a36517122c0325fd31db5ba6066d82ea

Modified Files
--------------
contrib/postgres_fdw/postgres_fdw.c                |  22 ++---
src/backend/access/brin/brin.c                     |   8 +-
src/backend/access/gist/gistbuild.c                |  16 ++--
src/backend/catalog/objectaddress.c                |  30 +++---
src/backend/catalog/pg_constraint.c                |  32 +++----
src/backend/commands/extension.c                   |   8 +-
src/backend/commands/schemacmds.c                  |   4 +-
src/backend/commands/statscmds.c                   |   6 +-
src/backend/commands/tablecmds.c                   |  14 +--
src/backend/commands/trigger.c                     |  12 +--
src/backend/executor/nodeAgg.c                     |  16 ++--
src/backend/executor/nodeValuesscan.c              |   4 +-
src/backend/optimizer/path/equivclass.c            |   6 +-
src/backend/optimizer/plan/createplan.c            |  44 ++++-----
src/backend/partitioning/partdesc.c                |  12 +--
src/backend/statistics/dependencies.c              |  30 +++---
src/backend/statistics/extended_stats.c            |   6 +-
src/backend/storage/aio/read_stream.c              |  14 +--
src/backend/storage/buffer/bufmgr.c                |  14 +--
src/backend/utils/adt/jsonpath_exec.c              |  48 +++++-----
src/backend/utils/adt/pg_upgrade_support.c         |   6 +-
src/backend/utils/adt/timestamp.c                  |  20 ++--
src/backend/utils/adt/varlena.c                    |  20 ++--
src/backend/utils/cache/inval.c                    |  48 +++++-----
src/backend/utils/mmgr/freepage.c                  |  42 ++++-----
src/bin/pg_basebackup/pg_receivewal.c              |   6 +-
src/bin/pgbench/pgbench.c                          |  62 ++++++------
src/bin/psql/describe.c                            |  18 ++--
src/bin/psql/prompt.c                              |  39 ++++----
src/bin/psql/prompt.h                              |   2 +-
src/fe_utils/print.c                               |  22 ++---
src/include/lib/radixtree.h                        |  16 ++--
src/include/optimizer/paths.h                      |   2 +-
src/include/storage/sinval.h                       |   4 +-
.../ecpg/test/expected/pgtypeslib-num_test2.c      |   3 +-
src/interfaces/ecpg/test/pgtypeslib/num_test2.pgc  |   3 +-
src/interfaces/libpq/fe-connect.c                  |  12 ++-
src/interfaces/libpq/fe-secure-openssl.c           | 104 ++++++++++-----------
38 files changed, 393 insertions(+), 382 deletions(-)

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Heikki Linnakangas <hlinnaka@iki.fi>
Дата:
On 18/08/2026 13:31, Hüseyin Demir wrote:
> Hi,
> 
>> v5 resolved this as a dry run rather than a hard block.  --check
>> --initdb reports the initdb command it would run and validates the
>> preconditions (correct new-binary version, empty target directory, old
>> cluster reachable) without creating the new cluster. v6 keeps that,
>> and the docs now describe it.
> 
> Thanks, appreciated.

Hmm, so without this, when you run "pg_upgrade --check ...", it checks 
various things:

$ bin/pg_upgrade --check -d data -b bin  -D data-new
Performing Consistency Checks
-----------------------------
Checking cluster versions                                     ok
Checking database connection settings                         ok
Checking for unsupported encodings                            ok
Checking database user is the install user                    ok
Checking for prepared transactions                            ok
Checking oldestXID and oldestMultiXid consistency             ok
Checking for contrib/isn with bigint-passing mismatch         ok
Checking logical replication slots                            ok
Checking subscription state                                   ok
Checking data type usage                                      ok
Checking for objects affected by Unicode update               ok
Checking for presence of required libraries                   ok
Checking database user is the install user                    ok
Checking for prepared transactions                            ok
Checking new cluster tablespace directories                   ok

*Clusters are compatible*

With the --initdb option, it skips all those checks:

$ bin/pg_upgrade --check -d data -b bin  -D data-new2 --initdb  -B bin
Examining old cluster settings                                ok
Constructing new cluster initdb command                       ok
The following initdb command would be run to create the new cluster:
   bin/initdb -N -D data-new2 -U heikki --wal-segsize=16 
--data-checksums --encoding=UTF8 --locale-provider=libc 
--lc-collate=en_GB.UTF-8 --lc-ctype=en_GB.UTF-8
The new cluster would be created with settings matching the old cluster. 
  Run pg_upgrade --check afterward for the full compatibility check.

That's not good, that essentially means that --check doesn't work with 
the --initdb option. Some of the checks check that the new cluster is 
compatible with the old cluster, and assuming we run initdb correctly, 
there's no need to run those checks with --initdb. But we should still 
run all the checks we can on the old cluster.


Another little issue I noticed: Without the --new-bindir/-B option, it 
fails to find the binary:

$ bin/pg_upgrade --check -d data -b bin  -D data-new2 --initdb
could not find a "pg_upgrade" to execute

pg_upgrade: could not find own program executable
Failure, exiting

The new binary dir is not given, it's supposed to default to the same as 
the old one.

- Heikki


Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Heikki Linnakangas <hlinnaka@iki.fi>
Дата:
On 17/07/2026 15:10, Bohyun Lee wrote:
> -O: dropped support with --initdb entirely, as you originally suggested. 
> The partial "-c only" forwarding still broke on quoted values with 
> spaces. pg_upgrade now rejects -O + --initdb during option parsing, with 
> a test and a doc note.

> @@ -244,6 +249,17 @@ parseCommandLine(int argc, char *argv[])
>         if (optind < argc)
>                 pg_fatal("too many command-line arguments (first is \"%s\")", argv[optind]);
>  
> +       /*
> +        * -O passes options to the new cluster's postmaster, but with --initdb
> +        * the new cluster is created by initdb, which accepts a different option
> +        * set.  Rather than guess which -O options initdb also understands, reject
> +        * the combination and let the user create the cluster manually (without
> +        * --initdb) if they need postmaster-only options.
> +        */
> +       if (new_cluster.pgopts && user_opts.initdb_new_cluster)
> +               pg_fatal("options %s and %s cannot be used together",
> +                                "-O/--new-options", "--initdb");
> +
>         if (!user_opts.sync_method)
>                 user_opts.sync_method = pg_strdup("fsync");
>  

That doesn't sound right. There's indeed no reason think that initdb 
would accept the same options as postmaster, so it doesn't make sense to 
pass the -O options to initdb. But why forbid using -O with --initdb? I 
would imagine that the -O options are passed to the postmaster the same 
way with or without --initdb.

Is there a way to pass extra options to initdb? Or do you require the 
user to run initdb first or change the settings afterwards, if they want 
some additional settings?

> +       
> +        To discover the old cluster's encoding and locale,
> +         briefly starts the old server in
> +        binary-upgrade mode (which disables autovacuum) and stops it again
> +        before creating the new cluster.  If a compatibility check fails after
> +        the new cluster has been created, its data directory is removed
> +        automatically, so the upgrade can be retried without manual cleanup.
> +       

Hmm, how's that different from how it works without --initdb? pg_upgrade 
needs to launch the old cluster anyway to perform the compatibility 
checks, right?

- Heikki



Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Bohyun Lee <bohyun.lee@databricks.com>
Дата:
Вложения:
Hi Shihao and Heikki,

Thanks for testing and for the feedback.

1. Meson coverage — Shihao #1
> t/009_initdb_option.pl is not listed in
> src/bin/pg_upgrade/meson.build, so it never runs in a meson build.

Thanks, included t/009_initdb_option.pl in the Meson test list.

2. Group access — Shihao #2
> The group access setting from initdb -g is not copied from the old
> cluster.

The current patch now preserves the old cluster's group-access setting
by passing --allow-group-access to initdb when the old cluster allows
group access.

3. Missing -B — Shihao #3 and Heikki

Shihao Zhong wrote:

> resolve_new_bindir() is a nice cleanup, but the new callers pass
> os_info.progname.  That is get_progname(argv[0]), so it has no directory
> in it.  find_my_exec() then searches PATH instead of using argv[0].

Heikki Linnakangas wrote:

> Another little issue I noticed: Without the --new-bindir/-B option, it
> fails to find the binary:
>
> $ bin/pg_upgrade --check -d data -b bin  -D data-new2 --initdb
> could not find a "pg_upgrade" to execute

The current patch now passes the original argv[0] to
resolve_new_bindir(). Without -B or PGBINNEW, the new binary directory
defaults to the directory containing pg_upgrade, matching the existing
behavior.

4. Log location and cleanup — Shihao #4
> The <pgdata>.initdb_log directory is never deleted.  It stays after a
> successful run and after --check, and --retain has no effect on it.

The patch is updated so that it now writes --initdb logs under ./pg_upgrade_output.d
in the current working directory, using the normal cleanup and --retain
handling. Successful runs, including --check, remove their logs unless
--retain is given; failed runs keep them.
This avoids requiring write access to the new data directory's parent
just for logging. Logs stay outside the new data directory because
initdb requires it to be empty.

5. Existing empty target — Shihao #5
> A small note on the empty directory check.  It stops initdb from
> running where there are files, but the cleanup still calls rmtree() on
> the directory itself.  So a directory the operator made by hand is gone
> after a failed run.

The patch now leaves initialization-failure cleanup to initdb,
which preserves an operator-created directory. It removes the contents
of a pre-existing empty target or removes a directory it created, unless
--no-clean was requested. Initdb sets the directory permissions as
usual, and the original mode is not restored.
Once initialization succeeds, pg_upgrade leaves the new cluster in place
if any later pg_upgrade step fails, as it does for a manually
initialized cluster.

6. --check --initdb — Shihao #6 and Heikki

Shihao Zhong wrote:

> Two things about --check --initdb.  I saw that v5 turned this into a
> dry run instead of blocking it.  First, the live check problem Daniel
> reported for v2 is back.  With the old cluster running, plain --check
> says "Clusters are compatible", while --check --initdb says "There seems
> to be a postmaster servicing the old cluster".  Second, the dry run
> returns 0 for a cluster that cannot be upgraded.  With a regproc column
> in the old cluster, --check --initdb returns 0 and the real run returns
> 1.

Heikki Linnakangas wrote:

> That's not good, that essentially means that --check doesn't work with
> the --initdb option. Some of the checks check that the new cluster is
> compatible with the old cluster, and assuming we run initdb correctly,
> there's no need to run those checks with --initdb. But we should still
> run all the checks we can on the old cluster.

The current patch now allows --check --initdb to check a running old
server without stopping it. The new server must use a different port.

Additionally, the patch now initializes the new cluster and runs the
compatibility checks on both clusters for --check --initdb, without
performing the upgrade. Unsupported regproc columns cause the check to
fail.

--check --initdb retains the initialized new cluster whether the compatibility
checks succeed or fail. After a successful check, rerun pg_upgrade against that
cluster without --check, --initdb, or --initdb-options.

7. -O and extra initdb options — Heikki
> But why forbid using -O with --initdb? I
> would imagine that the -O options are passed to the postmaster the same
> way with or without --initdb.

Right, the patch now allows -O with --initdb and passes it only to the
new postmaster, using the existing server-option handling.

8. Source-server startup description — Heikki
> Hmm, how's that different from how it works without --initdb? pg_upgrade
> needs to launch the old cluster anyway to perform the compatibility
> checks, right?

Yes. With --initdb, the old cluster's encoding and locale are read
before initializing the new cluster. The current implementation now omits this
startup detail from the option documentation since the Notes already describe
temporary server starts.

9. Passing extra options to initdb — Heikki
> Is there a way to pass extra options to initdb? Or do you require the
> user to run initdb first or change the settings afterwards, if they want
> some additional settings?

The current patch now provides --initdb-options to pass additional
arguments to initdb, for example:

pg_upgrade --initdb \
  --initdb-options='-c huge_pages=off --waldir=/wal/new' ...

It requires --initdb and it can be repeated, preserving upstream initdb option
ordering.
Settings passed with initdb -c apply during initialization and remain in
postgresql.conf. Options that conflict with pg_upgrade's managed settings are
rejected, while other option values and paths are validated by initdb.

Github repo: https://github.com/LeeBohyun/postgres/tree/pg_upgrade_initdb

Best,
Bohyun

On Mon, Oct 5, 2026 at 2:49 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
On 17/07/2026 15:10, Bohyun Lee wrote:
> -O: dropped support with --initdb entirely, as you originally suggested.
> The partial "-c only" forwarding still broke on quoted values with
> spaces. pg_upgrade now rejects -O + --initdb during option parsing, with
> a test and a doc note.

> @@ -244,6 +249,17 @@ parseCommandLine(int argc, char *argv[])
>         if (optind < argc)
>                 pg_fatal("too many command-line arguments (first is \"%s\")", argv[optind]);
> 
> +       /*
> +        * -O passes options to the new cluster's postmaster, but with --initdb
> +        * the new cluster is created by initdb, which accepts a different option
> +        * set.  Rather than guess which -O options initdb also understands, reject
> +        * the combination and let the user create the cluster manually (without
> +        * --initdb) if they need postmaster-only options.
> +        */
> +       if (new_cluster.pgopts && user_opts.initdb_new_cluster)
> +               pg_fatal("options %s and %s cannot be used together",
> +                                "-O/--new-options", "--initdb");
> +
>         if (!user_opts.sync_method)
>                 user_opts.sync_method = pg_strdup("fsync");
> 

That doesn't sound right. There's indeed no reason think that initdb
would accept the same options as postmaster, so it doesn't make sense to
pass the -O options to initdb. But why forbid using -O with --initdb? I
would imagine that the -O options are passed to the postmaster the same
way with or without --initdb.

Is there a way to pass extra options to initdb? Or do you require the
user to run initdb first or change the settings afterwards, if they want
some additional settings?

> +       <para>
> +        To discover the old cluster's encoding and locale,
> +        <option>--initdb</option> briefly starts the old server in
> +        binary-upgrade mode (which disables autovacuum) and stops it again
> +        before creating the new cluster.  If a compatibility check fails after
> +        the new cluster has been created, its data directory is removed
> +        automatically, so the upgrade can be retried without manual cleanup.
> +       </para>

Hmm, how's that different from how it works without --initdb? pg_upgrade
needs to launch the old cluster anyway to perform the compatibility
checks, right?

- Heikki

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Heikki Linnakangas <hlinnaka@iki.fi>
Дата:
On 07/10/2026 20:44, Bohyun Lee wrote:
> *2. Group access — Shihao #2*
>  > The group access setting from initdb -g is not copied from the old
>  > cluster.
> 
> The current patch now preserves the old cluster's group-access setting
> by passing --allow-group-access to initdb when the old cluster allows
> group access.

I don't think we should do this. If the user wants the new cluster to 
have group access, use --initdb-options='--allow-group-access'. There's 
no magic for that before this patch either, the user needs to use 
"initdb --allow_group_access" if they want the new cluster to have group 
access.

The user could've made other changes to the permissions too, like change 
the group that owns the data dir etc. If we tried to accurately copy all 
the old permissions, that seems like an endless task.

> *4. Log location and cleanup — Shihao #4*
>  > The .initdb_log directory is never deleted.  It stays after a
>  > successful run and after --check, and --retain has no effect on it.
> 
> The patch is updated so that it now writes --initdb logs under ./ 
> pg_upgrade_output.d
> in the current working directory, using the normal cleanup and --retain
> handling. Successful runs, including --check, remove their logs unless
> --retain is given; failed runs keep them.

Why not write the logs under the new data directory, just like without 
--initdb?

> This avoids requiring write access to the new data directory's parent
> just for logging. Logs stay outside the new data directory because
> initdb requires it to be empty.

But then you need write access to the current directory instead.

> Heikki Linnakangas wrote:
> 
>  > That's not good, that essentially means that --check doesn't work with
>  > the --initdb option. Some of the checks check that the new cluster is
>  > compatible with the old cluster, and assuming we run initdb correctly,
>  > there's no need to run those checks with --initdb. But we should still
>  > run all the checks we can on the old cluster.
> 
> The current patch now allows --check --initdb to check a running old
> server without stopping it.

Ok

> The new server must use a different port.

Why?

> Additionally, the patch now initializes the new cluster and runs the
> compatibility checks on both clusters for --check --initdb, without
> performing the upgrade. Unsupported regproc columns cause the check to
> fail.

There should be no need to perform checks on the new cluster. We're 
creating it, so it should be created with all the right settings.

Hmm, although I guess a user-supplied --initdb-options could contain 
bogus options, or options that make the new cluster incompatible in some 
way. Ok, I guess it's good to check for that somehow.

> --check --initdb retains the initialized new cluster whether the 
> compatibility
> checks succeed or fail. After a successful check, rerun pg_upgrade 
> against that
> cluster without --check, --initdb, or --initdb-options.

Does that mean that if you run "pg_upgrade .. --check --initdb" twice, 
it will fail the second time because the new cluster already exists? 
That doesn't sound nice.

> *9. Passing extra options to initdb — Heikki*
>  > Is there a way to pass extra options to initdb? Or do you require the
>  > user to run initdb first or change the settings afterwards, if they want
>  > some additional settings?
> 
> The current patch now provides --initdb-options to pass additional
> arguments to initdb, for example:
> 
> pg_upgrade --initdb \
>    --initdb-options='-c huge_pages=off --waldir=/wal/new' ...
> 
> It requires --initdb and it can be repeated, preserving upstream initdb 
> option
> ordering.

Ok.

> Settings passed with initdb -c apply during initialization and remain in
> postgresql.conf. Options that conflict with pg_upgrade's managed 
> settings are
> rejected, while other option values and paths are validated by initdb.

The code for those compatibility checks seem overly complicated. We 
shouldn't need to duplicate all the initdb options in pg_upgrade just 
for this. What if we just didn't do any of that validation and passed 
the initdb options directly to initdb, and let initdb fail if they're 
incorrect?

If we must have some cross-checking like this, then I'd suggest adding a 
hidden "initdb --for-pg-upgrade" option or something, to let initdb know 
that it's been called as part of pg_ugprade, and have initdb itself 
reject the options that are not allowed with pg_upgrade. Seems like less 
duplication that way. But please see if we can get away without any 
validation like this first.

- Heikki



Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
harrisontbooth@gmail.com
Дата:
Вложения:
I rebased the Windows test cleanup onto your branch at 170df44c. It
detaches the two test-owned WAL junctions only after the test passes
and preserves them for failed tests or PG_TEST_NOCLEAN.

The full test passed on native ARM64 and emulated x64. I also verified
both architectures retain the junctions after the existing long-path
failures and with PG_TEST_NOCLEAN=0. Those path failures reproduce
without this cleanup change.

The attached mbox contains your five branch commits unchanged,
followed by the cleanup supplement. It applies with git am on the
recorded author base and master 6b3f8857; runtime testing used the
recorded author-based builds.

Let me know if this cleanup approach works for you.

Best,

Harrison

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Bohyun Lee <bohyun.lee@databricks.com>
Дата:
Вложения:
> Let me know if this cleanup approach works for you.

Thanks, Harrison, for testing on both Windows architectures. 
The cleanup approach looks good to me. I’ve integrated your supplement, 
and the tests also pass on Linux and macOS.


Heikki, thanks again for the review and the comments. 
Also incorporated the order change you proposed in this patch.

> I don't think we should do this. If the user wants the new cluster to

> have group access, use --initdb-options='--allow-group-access'. There's

> no magic for that before this patch either, the user needs to use

> “initdb --allow_group_access" if they want the new cluster to have group

> access.


Right, removed the automatic inheritance of the old cluster's group-access setting.

The new cluster now uses initdb's default permissions, and users can enable group 

access explicitly with --initdb-options='--allow-group-access'.

This way, the behavior of manual initdb --allow-group-access + pg_upgrade execution 

remains the same with pg_upgrade --initdb --initdb-options='--allow-group-access'.

Updated the documentation and the existing test expectations as well.



> Why not write the logs under the new data directory, just like without

> --initdb?


Agreed, the normal logs should end up in the new datadir, as they do without --initdb.

The patch captures early server and initdb output under 

./pg_upgrade_output.d/<timestamp>/ in the current working directory, 

because initdb requires an empty target directory.

But once initdb succeeds, it removes the temporary logs unless --retain is specified.

Subsequent output is written under 

<new datadir>/pg_upgrade_output.d/<timestamp>/ using the existing cleanup and --retain handling.

If initdb fails, pg_upgrade reports initialization failure and points to the retained log.


> But then you need write access to the current directory instead.


pg_upgrade already requires a writable current directory, even without --initdb.

So, logging to a temporary location therefore add no new requirement for write 

access to that directory.



>> The new server must use a different port.

> Why?


The previous statement was an overclaim.

To clarify, not every --check --initdb invocation requires different ports.

Different ports are required only when checking a running old server.

This is an existing pg_upgrade requirement, unchanged by --initdb.

Offline checks can use the same port.



>> --check --initdb retains the initialized new cluster whether the

>> compatibility

>> checks succeed or fail. After a successful check, rerun pg_upgrade

>> against that

>> cluster without --check, --initdb, or --initdb-options.

> Does that mean that if you run "pg_upgrade .. --check --initdb" twice,

> it will fail the second time because the new cluster already exists?

> That doesn't sound nice.


Right, repeated checks now reuse the existing initialized cluster rather than failing.

When it is reused, pg_upgrade prints a notice and skips --initdb-options, 

while synchronization still honors --no-sync and --sync-method.

So the behavior is now as follows:

  • Empty or missing target: initialize it, then check.

  • Already initialized target: skip initialization and run the existing checks.

  • Invalid target or one containing user tables: reject it through those checks.

  


> What if we just didn't do any of that validation and passed

> the initdb options directly to initdb, and let initdb fail if they're

> incorrect?

>

> If we must have some cross-checking like this, then I'd suggest adding a

> hidden "initdb --for-pg-upgrade" option or something, to let initdb know

> that it's been called as part of pg_ugprade, and have initdb itself

> reject the options that are not allowed with pg_upgrade. 


Agreed, removed pg_upgrade's semantic validation and the duplicated option table.

It still splits and quotes --initdb-options and appends them after the generated 

initdb arguments, so initdb validates the options and overlapping options can 

override those arguments.pg_upgrade runs its existing checks on the resulting cluster.



While reworking the patch, I found additional issue regarding durability.

Previously, pg_upgrade --check --initdb returned successfully without syncing 

the retained new cluster to disk. The current patch's --check --initdb now 

synchronizes the new cluster after stopping its server before the successful return 

by using initdb --sync-only and honoring --no-sync and --sync-method.


Best regards,

Bohyun


On Wed, Oct 14, 2026 at 7:58 AM <harrisontbooth@gmail.com> wrote:
I rebased the Windows test cleanup onto your branch at 170df44c. It
detaches the two test-owned WAL junctions only after the test passes
and preserves them for failed tests or PG_TEST_NOCLEAN.

The full test passed on native ARM64 and emulated x64. I also verified
both architectures retain the junctions after the existing long-path
failures and with PG_TEST_NOCLEAN=0. Those path failures reproduce
without this cleanup change.

The attached mbox contains your five branch commits unchanged,
followed by the cleanup supplement. It applies with git am on the
recorded author base and master 6b3f8857; runtime testing used the
recorded author-based builds.

Let me know if this cleanup approach works for you.

Best,

Harrison

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Heikki Linnakangas <hlinnaka@iki.fi>
Дата:
On 14/10/2026 11:32, Bohyun Lee wrote:
>> Why not write the logs under the new data directory, just like without
>> --initdb?
> 
> Agreed, the normal logs should end up in the new datadir, as they do 
> without --initdb.
> 
> The patch captures early server and initdb output under
> 
> ./pg_upgrade_output.d// in the current working directory,
> 
> because initdb requires an empty target directory.
> 
> But once initdb succeeds, it removes the temporary logs unless --retain 
> is specified.
> 
> Subsequent output is written under
> 
> /pg_upgrade_output.d// using the existing 
> cleanup and --retain handling.
> 
> If initdb fails, pg_upgrade reports initialization failure and points to 
> the retained log.

That sounds weird. To find the logs, you have to check both places; they 
might be under ./pg_upgrade_output.d/ or /pg_upgrade_output.d/ depending on the stage that it failed at. 
Also, if initdb succeeds but the operation fails later, there's no trace 
of the successful initdb in /pg_upgrade_output.d/, those 
logs are lost even though the logs for other stages are there.

I see why you did it that way; there's a chicken-and-egg problem. In 
order to log the initdb invocation, the log dir must be created before 
initdb is launched, but initdb errors out if the target directory is not 
empty.

We could modify initdb so that it ignores "pg_upgrade_output.d" and 
considers the target dir as empty as long as nothing else exists. 
Another idea is to run initdb in a subdir first, and rename it into the 
right place later (see below).

>> But then you need write access to the current directory instead.
> 
> pg_upgrade already requires a writable current directory, even without 
> --initdb.
> 
> So, logging to a temporary location therefore add no new requirement
> for write access to that directory.

Oh, I see. (But see previous point).

>>> --check --initdb retains the initialized new cluster whether the
>>> compatibility
>>> checks succeed or fail. After a successful check, rerun pg_upgrade
>>> against that
> 
>>> cluster without --check, --initdb, or --initdb-options.
> 
>> Does that mean that if you run "pg_upgrade .. --check --initdb" twice,
>> it will fail the second time because the new cluster already exists?
> 
>> That doesn't sound nice.
> 
> Right, repeated checks now reuse the existing initialized cluster rather 
> than failing.

That's surprising too. I would expect "--check" to leave the directories 
unmodified. Except for the logs.


Putting all this together, I think this would work (in non-check mode):

1. Check that "" is empty or non-existent, like initdb does. 
(An existing "/pg_upgrade_output.d" dir is acceptable though)
2. Create "/pg_upgrade_output.d" and the log files
3. Run "initdb /temporary_data_dir"
4. Move all contents of "/temporary_data_dir" to ""
5. Delete the now-empty "/temporary_data_dir" directory.

In --check mode, you can go through the same motions, except that in 
step 4 you can just delete "temporary_data_dir".

One detail that's not quite right with this is the --allow-group-access 
again. With the above, the "" would not have the right 
permissions, because pg_upgrade doesn't know the right permissions in 
step 1 yet. Maybe you could fix that up around steps 4-5: read the 
permissions of "/temporary_data_dir" and copy them to "".


The other approach I mentioned above is to teach initdb to ignore 
"pg_upgrade_output.d". Then the sequence would be simply:

1. Create "/pg_upgrade_output.d" and the log files
2. Run "initdb -D ". It ignores "pg_upgrade_output.d", but will 
complain if there's anything else in ""

In --check mode, it's a little more complicated though:

1. Create "/pg_upgrade_output.d" and the log files
2. Run "initdb -D ". It ignores "pg_upgrade_output.d", but will 
complain if there's anything else in ""
3. Delete everything under "" except for "pg_upgrade_output.d"

Step 3 is a bit scary. So we could still use a temporary dir in --check 
mode:

1. Create "/pg_upgrade_output.d" and the log files
2. Run "initdb -D /temporary_data_dir".
3. Delete "/temporary_data_dir"

The downside of that is that the --check mode diverges more from the 
real mode, so there's some risk of things working in --check mode, but 
failing when you run it for real.

I'm not sure which of these approaches I like the best. What do you 
think? But I think the overall goals are:

- the logs always go under "/pg_upgrade_output.d"
- the logs contain the initdb output too
- --check mode leaves the target dir unmodified


- Heikki


Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Bohyun Lee <bohyun.lee@databricks.com>
Дата:
Вложения:
thanks again for the review, Heikki.

> To find the logs, you have to check both places
>
> Also, if initdb succeeds but the operation fails later, there's no trace
> of the successful initdb in <new datadir>/pg_upgrade_output.d/

All logs, including initdb output, now go under
<new datadir>/pg_upgrade_output.d/<timestamp>/ from the beginning. To make
this possible, I followed your suggestion to let initdb accept
pg_upgrade_output.d in an otherwise empty target.

I considered using a temporary datadir, but initializing directly in the
final data directory seemed less intrusive. This way, moving initialized
files or rewriting configuration paths can be avoided and lets both upgrade
and check modes use the final data directory throughout.

The initdb output remains there if a later stage fails. Successful runs
use the existing log cleanup and --retain handling.


> We could modify initdb so that it ignores "pg_upgrade_output.d" and
> considers the target dir as empty as long as nothing else exists.

This approach is now implemented for both upgrade and check modes, with a
hidden --for-pg-upgrade=<state-file> option used only by pg_upgrade.
The state file stores cleanup information and is removed after a successful
run unless --retain is specified.

Without this option, initdb would reject the target because pg_upgrade has
already created pg_upgrade_output.d inside it. This option lets initdb
accept a target containing only pg_upgrade_output.d and preserve that log
directory if initialization fails. Other initialized files in the new
datadir follow initdb's normal failure cleanup.

The state file contains the external WAL path, whether the directory
existed before initdb, its original permissions if it existed, and its
nearest existing parent if new directories were needed. pg_upgrade uses
this information to remove generated WAL and restore the prior directory
state after --check, without duplicating initdb's option handling.

When running pg_upgrade --initdb, with or without --check, the workflow is:

phase 1. Check that new_datadir is empty or missing, allowing an existing
   pg_upgrade_output.d directory. Create logs under
   new_datadir/pg_upgrade_output.d/<timestamp>/.

phase 2. Run initdb directly in new_datadir with
   --for-pg-upgrade=<state-file>. Here, <state-file> is
   new_datadir/pg_upgrade_output.d/<timestamp>/log/initdb_state.

phase 3.1. With --check: read the state file and run the normal compatibility
     checks against the initialized cluster. After confirming shutdown,
     remove initialized data and generated WAL, and restore prior target
     and WAL directory existence and permissions. Cleanup runs whether
     the compatibility checks succeed or fail.

phase 3.2. Without --check: run the normal compatibility checks against the
     initialized cluster. If they pass, continue upgrading.


> I would expect "--check" to leave the directories unmodified. Except
> for the logs.

Agreed, after successful initialization, --check removes the initialized
data, except for the log, which is placed inside
new_datadir/pg_upgrade_output.d/<timestamp>/.


> With the above, the "<new dir>" would not have the right permissions

In the current patch, initdb now sets the target permissions itself, using its 
defaults or an explicit --allow-group-access. pg_upgrade applies matching permissions
to newly created parents and logs, leaving existing parents unchanged.


> Step 3 is a bit scary. So we could still use a temporary dir in --check
> mode:
>
> 1. Create "<new dir>/pg_upgrade_output.d" and the log files
> 2. Run "initdb -D <new dir>/temporary_data_dir".
> 3. Delete "<new dir>/temporary_data_dir"

Right, the current patch's cleanup now requires successful initialization
and passing data-directory checks. If the state file cannot be read or shutdown
is not confirmed, pg_upgrade reports failure and retains the cluster and logs.
Data or WAL removal errors also report failure and preserve logs and any remaining data.
When cleanup completes, the workflow meets all three goals you mentioned.

Additional tests were conducted besides the TAP tests
to ensure that running initdb manually followed by pg_upgrade produces 
the same state after initialization and upgrade as pg_upgrade --initdb 
with the same options. The test script is not included in the patch but is available in the GitHub repository:
https://github.com/LeeBohyun/postgres/blob/pg_upgrade_initdb/pg-upgrade-initdb-parity.py

Best regards,
Bohyun

On Wed, Oct 14, 2026 at 7:54 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
On 14/10/2026 11:32, Bohyun Lee wrote:
>> Why not write the logs under the new data directory, just like without
>> --initdb?
>
> Agreed, the normal logs should end up in the new datadir, as they do
> without --initdb.
>
> The patch captures early server and initdb output under
>
> ./pg_upgrade_output.d/<timestamp>/ in the current working directory,
>
> because initdb requires an empty target directory.
>
> But once initdb succeeds, it removes the temporary logs unless --retain
> is specified.
>
> Subsequent output is written under
>
> <new datadir>/pg_upgrade_output.d/<timestamp>/ using the existing
> cleanup and --retain handling.
>
> If initdb fails, pg_upgrade reports initialization failure and points to
> the retained log.

That sounds weird. To find the logs, you have to check both places; they
might be under ./pg_upgrade_output.d/ or <new
datadir>/pg_upgrade_output.d/ depending on the stage that it failed at.
Also, if initdb succeeds but the operation fails later, there's no trace
of the successful initdb in <new datadir>/pg_upgrade_output.d/, those
logs are lost even though the logs for other stages are there.

I see why you did it that way; there's a chicken-and-egg problem. In
order to log the initdb invocation, the log dir must be created before
initdb is launched, but initdb errors out if the target directory is not
empty.

We could modify initdb so that it ignores "pg_upgrade_output.d" and
considers the target dir as empty as long as nothing else exists.
Another idea is to run initdb in a subdir first, and rename it into the
right place later (see below).

>> But then you need write access to the current directory instead.
>
> pg_upgrade already requires a writable current directory, even without
> --initdb.
>
> So, logging to a temporary location therefore add no new requirement
> for write access to that directory.

Oh, I see. (But see previous point).

>>> --check --initdb retains the initialized new cluster whether the
>>> compatibility
>>> checks succeed or fail. After a successful check, rerun pg_upgrade
>>> against that
>
>>> cluster without --check, --initdb, or --initdb-options.
>
>> Does that mean that if you run "pg_upgrade .. --check --initdb" twice,
>> it will fail the second time because the new cluster already exists?
>
>> That doesn't sound nice.
>
> Right, repeated checks now reuse the existing initialized cluster rather
> than failing.

That's surprising too. I would expect "--check" to leave the directories
unmodified. Except for the logs.


Putting all this together, I think this would work (in non-check mode):

1. Check that "<new dir>" is empty or non-existent, like initdb does.
(An existing "<new dir>/pg_upgrade_output.d" dir is acceptable though)
2. Create "<new dir>/pg_upgrade_output.d" and the log files
3. Run "initdb <new dir>/temporary_data_dir"
4. Move all contents of "<new dir>/temporary_data_dir" to "<new dir>"
5. Delete the now-empty "<new dir>/temporary_data_dir" directory.

In --check mode, you can go through the same motions, except that in
step 4 you can just delete "temporary_data_dir".

One detail that's not quite right with this is the --allow-group-access
again. With the above, the "<new dir>" would not have the right
permissions, because pg_upgrade doesn't know the right permissions in
step 1 yet. Maybe you could fix that up around steps 4-5: read the
permissions of "<new dir>/temporary_data_dir" and copy them to "<new dir>".


The other approach I mentioned above is to teach initdb to ignore
"pg_upgrade_output.d". Then the sequence would be simply:

1. Create "<new dir>/pg_upgrade_output.d" and the log files
2. Run "initdb -D <new dir>". It ignores "pg_upgrade_output.d", but will
complain if there's anything else in "<new dir>"

In --check mode, it's a little more complicated though:

1. Create "<new dir>/pg_upgrade_output.d" and the log files
2. Run "initdb -D <new dir>". It ignores "pg_upgrade_output.d", but will
complain if there's anything else in "<new dir>"
3. Delete everything under "<new dir>" except for "pg_upgrade_output.d"

Step 3 is a bit scary. So we could still use a temporary dir in --check
mode:

1. Create "<new dir>/pg_upgrade_output.d" and the log files
2. Run "initdb -D <new dir>/temporary_data_dir".
3. Delete "<new dir>/temporary_data_dir"

The downside of that is that the --check mode diverges more from the
real mode, so there's some risk of things working in --check mode, but
failing when you run it for real.

I'm not sure which of these approaches I like the best. What do you
think? But I think the overall goals are:

- the logs always go under "<new dir>/pg_upgrade_output.d"
- the logs contain the initdb output too
- --check mode leaves the target dir unmodified


- Heikki

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
shihao zhong <zhong950419@gmail.com>
Дата:

Hi Bohyun,

Thanks for v9. My points on v6 are all handled. Two new ones.

1. ICU rules are not copied. If the old template0 has daticurules,
the new one has none and the upgrade still succeeds. A small fix is
attached.

2. If a real run fails in the checks, the new cluster stays, and the
same command then fails with "new cluster data directory is not
empty". Nothing has been copied yet at that point, so could it be
removed like in --check mode?

Thanks,
Shihao

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
Bohyun Lee <bohyun.lee@databricks.com>
Дата:
Вложения:
Hi Shihao,

Thanks for the review. Both points are addressed in v10.

> 1. ICU rules are not copied.

Thanks for catching that. The patch is updated so the upgrade now copies template0.daticurules.
I used a parameterized update so that multiline and long ICU rules are preserved as well.

> 2. If a real run fails in the checks, the new cluster stays,

A real --initdb run now uses the existing cleanup when compatibility
checks fail, just as in --check mode. The same command can then be retried.
Failures after the upgrade begins still retain the cluster.


Best regards,
Bohyun

On Thu, Oct 22, 2026 at 5:46 AM shihao zhong <zhong950419@gmail.com> wrote:

Hi Bohyun,

Thanks for v9. My points on v6 are all handled. Two new ones.

1. ICU rules are not copied. If the old template0 has daticurules,
the new one has none and the upgrade still succeeds. A small fix is
attached.

2. If a real run fails in the checks, the new cluster stays, and the
same command then fails with "new cluster data directory is not
empty". Nothing has been copied yet at that point, so could it be
removed like in --check mode?

Thanks,
Shihao

Re: [PATCH] pg_upgrade: add --initdb option to create the new cluster automatically

От:
shihao zhong <zhong950419@gmail.com>
Дата:
Hi Bohyun,

Thanks for v10.  I think this patch is LGTM

Thanks,
Shihao
FAQ