Re: Reproducible builds: genbki.pl and Gen_fmgrtab.pl

Christoph Berg <christoph.berg@credativ.de> writes:
> Re: Tom Lane 2017-12-16 <417.1513438031@sss.pgh.pa.us>
>> I think we're talking at cross-purposes.  I'm not saying we should not fix
>> this problem.  I'm saying that the proposed fix appears incomplete ...

> Grepping through the source, there are three places where $0 printed
> to files in regular operation (as opposed to being used in --help):

I poked around and found a few more.

> I believe the reason why we've only been seeing half of the problem
> yet is that the generated files are shipped with the tarballs, so it
> might be a timestamping issue determining if the scripts are
> re-executed.

Right; some parts of this problem would only materialize for you if you
needed to rebuild the generated files that are included in the tarball,
which should basically not be happening in normal packager builds.
Rather the risk is at our end: if we ever switched the tarball creation
process to be a VPATH build, then there'd be path dependencies in the
created tarballs.  That would be bad.

More generally, my concern here is not just that we fix this problem
but that it stays fixed.  If some individual scripts print $0 into
their output and it happens to not affect any built distribution files
today, it's still bad, because tomorrow somebody might copy that coding
pattern into someplace else where it matters more.  I think we need a
project policy that thou shalt not print $0 into generated files, period.

Also, experimenting with a VPATH build, I verified that such "helpful"
practices as printing $infile or @ARGV into the output file will also
create path dependencies.  So I think we need to lose those too.
It's not like they're adding any info you can't find out from the
Makefiles.

On the other hand, there is something we can do that will improve
matters: rather than just printing the base name of the script,
let's print its full relative path within the PG sources, eg instead
of Gen_fmgrtab.pl let's print src/backend/utils/Gen_fmgrtab.pl.
My thought here is that if you're not already intimately familiar
with a script you might not remember where it lives, the more so
if you're looking at a file that's been put into an installation
tree far away from where it was generated.  I see that this policy
was already followed in some places, just not in the ones that were
using the $0 shortcut.

In short, I propose the attached more-extensive patch.

Some of the files generated by these scripts, particularly the map
files generated by the src/backend/utils/mb/Unicode/ scripts, are
not just present in tarballs but are actually in our git repo.
So changing those scripts won't affect anything until/unless someone
updates the repo's generated files, which I've not done here and
don't feel much need to do.  I just want to establish a principle
that we don't print path-dependent info into generated files.

            regards, tom lane

diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl
index 256a9c9..e4a0b8b 100644
--- a/src/backend/catalog/genbki.pl
+++ b/src/backend/catalog/genbki.pl
@@ -340,7 +340,7 @@ print $schemapg <<EOM;
  *  *** DO NOT EDIT THIS FILE! ***
  *  ******************************
  *
- *  It has been GENERATED by $0
+ *  It has been GENERATED by src/backend/catalog/genbki.pl
  *
  *-------------------------------------------------------------------------
  */
diff --git a/src/backend/utils/Gen_fmgrtab.pl b/src/backend/utils/Gen_fmgrtab.pl
index ee89d50..26b428b 100644
--- a/src/backend/utils/Gen_fmgrtab.pl
+++ b/src/backend/utils/Gen_fmgrtab.pl
@@ -118,8 +118,7 @@ qq|/*-------------------------------------------------------------------------
  *    *** DO NOT EDIT THIS FILE! ***
  *    ******************************
  *
- *    It has been GENERATED by $0
- *    from $infile
+ *    It has been GENERATED by src/backend/utils/Gen_fmgrtab.pl
  *
  *-------------------------------------------------------------------------
  */
@@ -153,8 +152,7 @@ qq|/*-------------------------------------------------------------------------
  *    *** DO NOT EDIT THIS FILE! ***
  *    ******************************
  *
- *    It has been GENERATED by $0
- *    from $infile
+ *    It has been GENERATED by src/backend/utils/Gen_fmgrtab.pl
  *
  *-------------------------------------------------------------------------
  */
@@ -181,8 +179,7 @@ qq|/*-------------------------------------------------------------------------
  *    *** DO NOT EDIT THIS FILE! ***
  *    ******************************
  *
- *    It has been GENERATED by $0
- *    from $infile
+ *    It has been GENERATED by src/backend/utils/Gen_fmgrtab.pl
  *
  *-------------------------------------------------------------------------
  */
diff --git a/src/backend/utils/mb/Unicode/UCS_to_BIG5.pl b/src/backend/utils/mb/Unicode/UCS_to_BIG5.pl
index 05822b2..f73e2a1 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_BIG5.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_BIG5.pl
@@ -27,7 +27,7 @@
 use strict;
 use convutils;

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_BIG5.pl';

 # Load BIG5.TXT
 my $all = &read_source("BIG5.TXT");
diff --git a/src/backend/utils/mb/Unicode/UCS_to_EUC_CN.pl b/src/backend/utils/mb/Unicode/UCS_to_EUC_CN.pl
index f7f9496..63f5e1b 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_EUC_CN.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_EUC_CN.pl
@@ -16,7 +16,7 @@
 use strict;
 use convutils;

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_EUC_CN.pl';

 # Read the input

diff --git a/src/backend/utils/mb/Unicode/UCS_to_EUC_JIS_2004.pl b/src/backend/utils/mb/Unicode/UCS_to_EUC_JIS_2004.pl
index 2290031..4dac7d2 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_EUC_JIS_2004.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_EUC_JIS_2004.pl
@@ -10,7 +10,7 @@
 use strict;
 use convutils;

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_EUC_JIS_2004.pl';

 # first generate UTF-8 --> EUC_JIS_2004 table

diff --git a/src/backend/utils/mb/Unicode/UCS_to_EUC_JP.pl b/src/backend/utils/mb/Unicode/UCS_to_EUC_JP.pl
index 5db663f..77ce273 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_EUC_JP.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_EUC_JP.pl
@@ -14,7 +14,7 @@
 use strict;
 use convutils;

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_EUC_JP.pl';

 # Load JIS0212.TXT
 my $jis0212 = &read_source("JIS0212.TXT");
diff --git a/src/backend/utils/mb/Unicode/UCS_to_EUC_KR.pl b/src/backend/utils/mb/Unicode/UCS_to_EUC_KR.pl
index f0d978d..8e0126c 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_EUC_KR.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_EUC_KR.pl
@@ -19,7 +19,7 @@
 use strict;
 use convutils;

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_EUC_KR.pl';

 # Load the source file.

diff --git a/src/backend/utils/mb/Unicode/UCS_to_EUC_TW.pl b/src/backend/utils/mb/Unicode/UCS_to_EUC_TW.pl
index 0bfcbd5..61013e3 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_EUC_TW.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_EUC_TW.pl
@@ -20,7 +20,7 @@
 use strict;
 use convutils;

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_EUC_TW.pl';

 my $mapping = &read_source("CNS11643.TXT");

diff --git a/src/backend/utils/mb/Unicode/UCS_to_GB18030.pl b/src/backend/utils/mb/Unicode/UCS_to_GB18030.pl
index 4469cc7..e9f816c 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_GB18030.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_GB18030.pl
@@ -16,7 +16,7 @@
 use strict;
 use convutils;

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_GB18030.pl';

 # Read the input

diff --git a/src/backend/utils/mb/Unicode/UCS_to_JOHAB.pl b/src/backend/utils/mb/Unicode/UCS_to_JOHAB.pl
index 7c6f526..be10d52 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_JOHAB.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_JOHAB.pl
@@ -18,7 +18,7 @@
 use strict;
 use convutils;

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_JOHAB.pl';

 # Load the source file.

diff --git a/src/backend/utils/mb/Unicode/UCS_to_SHIFT_JIS_2004.pl
b/src/backend/utils/mb/Unicode/UCS_to_SHIFT_JIS_2004.pl
index d1b36c0..98a6ee7 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_SHIFT_JIS_2004.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_SHIFT_JIS_2004.pl
@@ -12,7 +12,7 @@ use convutils;

 # first generate UTF-8 --> SHIFT_JIS_2004 table

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_SHIFT_JIS_2004.pl';

 my $in_file = "sjis-0213-2004-std.txt";

diff --git a/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl b/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
index 7460964..cc1edcc 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
@@ -13,7 +13,7 @@
 use strict;
 use convutils;

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_SJIS.pl';

 my $mapping = read_source("CP932.TXT");

diff --git a/src/backend/utils/mb/Unicode/UCS_to_UHC.pl b/src/backend/utils/mb/Unicode/UCS_to_UHC.pl
index 8d99ca3..640a2ec 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_UHC.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_UHC.pl
@@ -16,7 +16,7 @@
 use strict;
 use convutils;

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_UHC.pl';

 # Read the input

diff --git a/src/backend/utils/mb/Unicode/UCS_to_most.pl b/src/backend/utils/mb/Unicode/UCS_to_most.pl
index 1c3922e..2d69748 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_most.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_most.pl
@@ -18,7 +18,7 @@
 use strict;
 use convutils;

-my $this_script = $0;
+my $this_script = 'src/backend/utils/mb/Unicode/UCS_to_most.pl';

 my %filename = (
     'WIN866'     => 'CP866.TXT',
diff --git a/src/bin/psql/create_help.pl b/src/bin/psql/create_help.pl
index cedb767..9fa1855 100644
--- a/src/bin/psql/create_help.pl
+++ b/src/bin/psql/create_help.pl
@@ -51,8 +51,7 @@ print $hfile_handle "/*
  * *** Do not change this file by hand. It is automatically
  * *** generated from the DocBook documentation.
  *
- * generated by
- *     $^X $0 @ARGV
+ * generated by src/bin/psql/create_help.pl
  *
  */

@@ -76,8 +75,7 @@ print $cfile_handle "/*
  * *** Do not change this file by hand. It is automatically
  * *** generated from the DocBook documentation.
  *
- * generated by
- *     $^X $0 @ARGV
+ * generated by src/bin/psql/create_help.pl
  *
  */

@@ -131,7 +129,7 @@ foreach my $file (sort readdir DIR)
         my $nl_count = () = $cmdsynopsis =~ /\n/g;

         $cmdsynopsis =~ m!</>!
-          and die "$0:$file: null end tag not supported in synopsis\n";
+          and die "$0: $file: null end tag not supported in synopsis\n";
         $cmdsynopsis =~ s/%/%%/g;

         while ($cmdsynopsis =~ m!<(\w+)[^>]*>(.+?)</\1[^>]*>!)
diff --git a/src/pl/plperl/text2macro.pl b/src/pl/plperl/text2macro.pl
index e681fca..27c6ef7 100644
--- a/src/pl/plperl/text2macro.pl
+++ b/src/pl/plperl/text2macro.pl
@@ -40,7 +40,7 @@ die "No text files specified"
 print qq{
 /*
  * DO NOT EDIT - THIS FILE IS AUTOGENERATED - CHANGES WILL BE LOST
- * Written by $0 from @ARGV
+ * Generated by src/pl/plperl/text2macro.pl
  */
 };

diff --git a/src/test/perl/TestLib.pm b/src/test/perl/TestLib.pm
index 6019040..72826d5 100644
--- a/src/test/perl/TestLib.pm
+++ b/src/test/perl/TestLib.pm
@@ -66,7 +66,7 @@ BEGIN
     delete $ENV{PGPORT};
     delete $ENV{PGHOST};

-    $ENV{PGAPPNAME} = $0;
+    $ENV{PGAPPNAME} = basename($0);

     # Must be set early
     $windows_os = $Config{osname} eq 'MSWin32' || $Config{osname} eq 'msys';