Обсуждение: many animals are running old clients

Animal owners,


The majority of buildfarm animals are not on the latest client code 
release. Version 8 was released more than 3 months ago. There are 26 
animals on release 7 and 43 on even older releases, in one case on a 
release going back 4 years. See below for the lists of delinquents.


If this situation doesn't improve I'm going to have to reinstitute the 
server filters which will reject very old clients.


I do my best to keep things backwards compatible, so that usually an 
upgrade is simply a matter of unpacking the release and you're done. I 
don't think asking people to do that once every few months is terribly 
onerous. It's a five minute job.


cheers


andrew


    sysname    |         owner_email         | script_version
--------------+-----------------------------+----------------
  alewife      | mark@2ndQuadrant.com        | 007000
  conchuela    | mikael.kjellstrom@gmail.com | 007000
  desmoxytes   | andres@anarazel.de          | 007000
  dory         | packagers@crunchydata.com   | 007000
  dragonet     | andres@anarazel.de          | 007000
  flaviventris | andres@anarazel.de          | 007000
  grison       | mikael.kjellstrom@gmail.com | 007000
  guaibasaurus | stefan@kaltenbrunner.cc     | 007000
  hamerkop     | buildfarm@sraoss.co.jp      | 007000
  hornet       | noah@leadboat.com           | 007000
  idiacanthus  | andres@anarazel.de          | 007000
  komodoensis  | andres@anarazel.de          | 007000
  lapwing      | pgbuildfarm@rjuju.net       | 007000
  loach        | mikael.kjellstrom@gmail.com | 007000
  macaque      | jgh@wizmail.org             | 007000
  mandrill     | noah@leadboat.com           | 007000
  mayfly       | clarenceho@gmail.com        | 007000
  petalura     | andres@anarazel.de          | 007000
  phycodurus   | andres@anarazel.de          | 007000
  pogona       | andres@anarazel.de          | 007000
  serinus      | andres@anarazel.de          | 007000
  sidewinder   | mikael.kjellstrom@gmail.com | 007000
  sungazer     | noah@leadboat.com           | 007000
  tern         | noah@leadboat.com           | 007000
  topminnow    | noah@leadboat.com           | 007000
  xenodermus   | andres@anarazel.de          | 007000
(26 rows)


     sysname    |         owner_email         | script_version
---------------+-----------------------------+----------------
  chipmunk      | heikki.linnakangas@iki.fi   | 004013
  gharial       | cm@enterprisedb.com         | 004015
  dunlin        | pgbuildfarm@jdrake.com      | 004015
  protosciurus  | dpage@pgadmin.org           | 004015
  anole         | cm@enterprisedb.com         | 004015
  koreaceratops | hcjeon@bitnine.co.kr        | 004015
  damselfly     | dap@joyent.com              | 004015
  castoroides   | dpage@pgadmin.org           | 004015
  clam          | cm@enterprisedb.com         | 004016
  quokka        | cm@enterprisedb.com         | 004016
  grouse        | mark@2ndquadrant.com        | 004017
  coypu         | remi_zara@mac.com           | 004017
  spurfowl      | sfrost@snowman.net          | 004018
  curculio      | mikael.kjellstrom@gmail.com | 004018
  rover_firefly | Keith@omniti.com            | 004019
  thrips        | chris@chrullrich.net        | 004019
  jaguarundi    | chris@chrullrich.net        | 005000
  caiman        | rosset.filipe@gmail.com     | 005000
  calliphoridae | andres@anarazel.de          | 005000
  chub          | bernd.helmle@credativ.de    | 005000
  culicidae     | andres@anarazel.de          | 005000
  dangomushi    | michael@otacoo.com          | 005000
  francolin     | andres@anarazel.de          | 005000
  gull          | clarenceho@gmail.com        | 005000
  handfish      | rosset.filipe@gmail.com     | 005000
  hyrax         | schmiddy@gmail.com          | 005000
  mantid        | tcook@blackducksoftware.com | 005000
  mereswine     | clarenceho@gmail.com        | 005000
  mylodon       | andres@anarazel.de          | 005000
  piculet       | andres@anarazel.de          | 005000
  skink         | andres@anarazel.de          | 005000
  whelk         | chris@chrullrich.net        | 005000
  woodlouse     | chris@chrullrich.net        | 005000
  rhinoceros    | mail@joeconway.com          | 006000
  vulpes        | mark@2ndQuadrant.com        | 006001
  sprat         | mark@2ndQuadrant.com        | 006001
  dhole         | mark@2ndQuadrant.com        | 006001
  takin         | mark@2ndQuadrant.com        | 006001
  locust        | remi_zara@mac.com           | 006001
  urocryon      | mark@2ndQuadrant.com        | 006001
  nudibranch    | mark@2ndQuadrant.com        | 006001
  mule          | gael@pilotsystems.net       | 006001
  cuon          | mark@2ndQuadrant.com        | 006001
(43 rows)


-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> The majority of buildfarm animals are not on the latest client code 
> release. Version 8 was released more than 3 months ago. There are 26 
> animals on release 7 and 43 on even older releases, in one case on a 
> release going back 4 years. See below for the lists of delinquents.

There's also a depressingly large number of animals that still aren't
building the REL_11_STABLE branch.  Some also have odd gaps in their
back-branch coverage.  If you don't use run_branches.pl to automate
branch selection, please also take the time to check that you are
building the right set of branches.

> If this situation doesn't improve I'm going to have to reinstitute the 
> server filters which will reject very old clients.

While I'd certainly encourage people to update, I'm not sure why
that's necessary?

            regards, tom lane

On 08/17/2018 05:51 PM, Tom Lane wrote:
> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>> The majority of buildfarm animals are not on the latest client code
>> release. Version 8 was released more than 3 months ago. There are 26
>> animals on release 7 and 43 on even older releases, in one case on a
>> release going back 4 years. See below for the lists of delinquents.
> There's also a depressingly large number of animals that still aren't
> building the REL_11_STABLE branch.  Some also have odd gaps in their
> back-branch coverage.  If you don't use run_branches.pl to automate
> branch selection, please also take the time to check that you are
> building the right set of branches.
>
>> If this situation doesn't improve I'm going to have to reinstitute the
>> server filters which will reject very old clients.
> While I'd certainly encourage people to update, I'm not sure why
> that's necessary?



Example: Some animals are still getting the Pre_run_port_check error. 
That doesn't even exist any more, it disappeared when the security 
enhancements went in.


cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2018-08-17 23:27, Andrew Dunstan wrote:

> The majority of buildfarm animals are not on the latest client code 
> release. Version 8 was released more than 3 months ago. There are 26 
> animals on release 7 and 43 on even older releases, in one case on a 
> release going back 4 years. See below for the lists of delinquents.

I've updated all my animals below to version 8.


>   conchuela    | mikael.kjellstrom@gmail.com | 007000
>   grison       | mikael.kjellstrom@gmail.com | 007000
>   loach        | mikael.kjellstrom@gmail.com | 007000
>   sidewinder   | mikael.kjellstrom@gmail.com | 007000
>   curculio      | mikael.kjellstrom@gmail.com | 004018

It might be worth to send out a email directly to the animal owners that 
a new version is out.  It's easy to miss if you don't follow the mailing 
lists.

Just a suggestion.

And regarding:

"I do my best to keep things backwards compatible, so that usually an 
upgrade is simply a matter of unpacking the release and you're done. I 
don't think asking people to do that once every few months is terribly 
onerous. It's a five minute job."

Every time I upgrade I need to go through all the .pl-files and change 
the path to where perl is located as different os/distros has perl at 
different paths and also it's not even consistent within the shipped files.

For example:

% head -n 1 *.pl
==> run_branches.pl <==
#!/usr/bin/perl

==> run_build.pl <==
#!/usr/bin/perl

==> run_web_txn.pl <==
#!/c/perl/bin/perl

==> setnotes.pl <==
#!/usr/bin/perl

==> update_personality.pl <==
#!/usr/bin/perl

I also go through the config-file line by line to see if anything is 
changed or added.

It's not a big deal though.  Keep up the good work!

/Mikael

On 08/18/2018 06:48 AM, Mikael Kjellström wrote:
>
> On 2018-08-17 23:27, Andrew Dunstan wrote:
>
>> The majority of buildfarm animals are not on the latest client code 
>> release. Version 8 was released more than 3 months ago. There are 26 
>> animals on release 7 and 43 on even older releases, in one case on a 
>> release going back 4 years. See below for the lists of delinquents.
>
> I've updated all my animals below to version 8.
>
>
>>   conchuela    | mikael.kjellstrom@gmail.com | 007000
>>   grison       | mikael.kjellstrom@gmail.com | 007000
>>   loach        | mikael.kjellstrom@gmail.com | 007000
>>   sidewinder   | mikael.kjellstrom@gmail.com | 007000
>>   curculio      | mikael.kjellstrom@gmail.com | 004018

Thanks.

>
> It might be worth to send out a email directly to the animal owners 
> that a new version is out.  It's easy to miss if you don't follow the 
> mailing lists.



Mail goes to the buildfarm-members list. That should be sufficiently 
specific. Having to email owners individually would be unpleasant.


>
> Just a suggestion.
>
> And regarding:
>
> "I do my best to keep things backwards compatible, so that usually an 
> upgrade is simply a matter of unpacking the release and you're done. I 
> don't think asking people to do that once every few months is terribly 
> onerous. It's a five minute job."
>
> Every time I upgrade I need to go through all the .pl-files and change 
> the path to where perl is located as different os/distros has perl at 
> different paths and also it's not even consistent within the shipped 
> files.
>
> For example:
>
> % head -n 1 *.pl
> ==> run_branches.pl <==
> #!/usr/bin/perl
>
> ==> run_build.pl <==
> #!/usr/bin/perl
>
> ==> run_web_txn.pl <==
> #!/c/perl/bin/perl
>
> ==> setnotes.pl <==
> #!/usr/bin/perl
>
> ==> update_personality.pl <==
> #!/usr/bin/perl
>
> I also go through the config-file line by line to see if anything is 
> changed or added.
>
> It's not a big deal though.  Keep up the good work!
>
> /Mikael


run_web_txn.pl is only ever used on Windows/msys, that's why its path is 
different. If you're not using one of those systems you can safely 
ignore it.

Would it help if I added a script to set the perl locations? 
Alternatively, we could switch to using /usr/bin/env, but I'm not sure 
how universal it is. And if you want to use a non-distro perl it 
possibly won't help anyway.


cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> On 08/18/2018 06:48 AM, Mikael Kjellström wrote:
>> Every time I upgrade I need to go through all the .pl-files and change 
>> the path to where perl is located as different os/distros has perl at 
>> different paths and also it's not even consistent within the shipped 
>> files.

> Would it help if I added a script to set the perl locations? 

Is it really necessary to touch the shebang lines at all?
I thought if you invoked the top script with
    perl-of-choice /path/to/run_build.pl ... args ...
then you'd be good.  If that's not so, maybe we could make it work?

>> I also go through the config-file line by line to see if anything is 
>> changed or added.

Yeah, this bit is the most time-consuming for me.  Not sure what's to
be done about it.

            regards, tom lane

On 08/18/2018 11:12 AM, Tom Lane wrote:
> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>> On 08/18/2018 06:48 AM, Mikael Kjellström wrote:
>>> Every time I upgrade I need to go through all the .pl-files and change
>>> the path to where perl is located as different os/distros has perl at
>>> different paths and also it's not even consistent within the shipped
>>> files.
>> Would it help if I added a script to set the perl locations?
> Is it really necessary to touch the shebang lines at all?
> I thought if you invoked the top script with
>     perl-of-choice /path/to/run_build.pl ... args ...
> then you'd be good.  If that's not so, maybe we could make it work?



Sadly it's not quite that simple. run_branches has this code:

    sub run_branch
    {
         my $branch = shift;
         my @args = ($run_build,
    PGBuild::Options::standard_option_list(), $branch);

         # Explicitly use perl from the path (and not this perl, so
    don't use $^X)
         # This script needs to run on Cygwin with non-cygwin perl if
    it's running
         # in tandem with AS/MinGW perl, since Cygwin perl doesn't honor
    locks
         # the samne way, and the global lock fails. But the build
    script needs
         # to run with the native perl, even on Cygwin, which it picks
    up from
         # the path. (Head exploding yet?).
         system("perl", @args);
         return;
    }

I could probably reasonably limit that so that everywhere but Cygwin we 
use the called perl. Windows is really the only place where we have to 
wrestle with multiple perls.



>
>>> I also go through the config-file line by line to see if anything is
>>> changed or added.
> Yeah, this bit is the most time-consuming for me.  Not sure what's to
> be done about it.
>
>             



It's in my list of things to consider. One of the things on the roadmap 
in my head is a possible switch to a YAML config file.

Perhaps I should publish in the release notes a diff of the sample 
config file against the previous release.

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2018-08-18 14:12, Andrew Dunstan wrote:

> Mail goes to the buildfarm-members list. That should be sufficiently 
> specific. Having to email owners individually would be unpleasant.

OK, yes I agree.


> run_web_txn.pl is only ever used on Windows/msys, that's why its path is 
> different. If you're not using one of those systems you can safely 
> ignore it.

OK, that makes sense.


> Would it help if I added a script to set the perl locations? 
> Alternatively, we could switch to using /usr/bin/env, but I'm not sure 
> how universal it is. And if you want to use a non-distro perl it 
> possibly won't help anyway.


Well, it's not that hard to go through all the .pl-files manually and 
change the path to where perl is located.

Would be nice to have a script that automatically detected the perl 
binary through using which or something like that but that wouldn't be 
portable and work on other OS:es like Windows, I guess.

I did get a problem with my NetBSD 7 animal not accepting the 
SSL-certificate (Let's Encrypt) that https://git.postgresql.org uses.

I got the error:
Git mirror failure:
Cloning into bare repository '/home/pgbf/buildroot/pgmirror.git'...
fatal: unable to access 
'https://git.postgresql.org/git/postgresql.git/': SSL certificate 
problem: unable to get local issuer certificate

I managed to solve that in NetBSD but following the following guide:

https://www.cambus.net/installing-ca-certificates-on-netbsd/

what that does is basically installing the mozilla-rootscerts on to the 
system and using it as default.

After that it worked fine again.  But I can imagine on older OS:es 
where you can't easily upgrade the CA-certs on the system it could be a 
big problem. HTTPS and the cert business is so broken it's not even 
funny imho.

/Mikael

On 08/18/2018 12:22 PM, Noah Misch wrote (offlist):
> On Sat, Aug 18, 2018 at 08:17:17AM -0400, Andrew Dunstan wrote:
>>
>> Is there anything I can do to make upgrade easier?
> The show_log.pl web page could highlight a particular run's config change,
> including version number changes, when there is such a change compared to the
> previous run.  If it's clear enough that folks are unlikely to confuse an
> upgrade-induced failure with a PG-commit-induced failure, I could stop checking
> for post-upgrade buildfarm failures.


What I've done is add this to the database in a way that makes it easily 
accessible to the web app, and added that info to the history page.

I might break that out into a separate history at some stage.


>
> It wouldn't make an upgrade easier, but the "Flags" column of show_status.pl
> could give an extra badge for being on the latest version.  That would motivate
> some folks.


Yes, I'll look at this, although I'm reluctant to make the dashboard any 
more busy.


cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hello,

I updated mine (mule), sorry for the lag on the update.

Regards,
--
Gaël Le Mignot - gael@pilotsystems.net
Pilot Systems - 82, rue de Pixérécourt - 75020 Paris
Tel : +33 1 44 53 05 55 - www.pilot-systems.net
Découvrez notre offre Cloud privé 100% infogéré - www.pilotsystems.net/cloud/

On Sun, Aug 19, 2018 at 03:04:58PM -0400, Andrew Dunstan wrote:
> On 08/18/2018 12:22 PM, Noah Misch wrote (offlist):
> >On Sat, Aug 18, 2018 at 08:17:17AM -0400, Andrew Dunstan wrote:
> >>Is there anything I can do to make upgrade easier?
> >The show_log.pl web page could highlight a particular run's config change,
> >including version number changes, when there is such a change compared to the
> >previous run.  If it's clear enough that folks are unlikely to confuse an
> >upgrade-induced failure with a PG-commit-induced failure, I could stop checking
> >for post-upgrade buildfarm failures.
> 
> What I've done is add this to the database in a way that makes it easily
> accessible to the web app, and added that info to the history page.

I think that wastes prime screen real estate.  If show_history.pl is going to
spend that space on something, I would choose commit hash.  But I liked it
better with nothing there.

The concept I had in mind was a "Config changed since last success" block
adjacent to the "Files changed since last success" block in show_log.pl.  It
might look like this:

- 'config_env' => { 'CC' => 'gcc' },
+ 'config_env' => { 'CC' => 'gcc -mips32r2' },
- 'script_version' => 'REL_7',
+ 'script_version' => 'REL_8',

Script version is a minor aspect, worth little by itself.  If you diff the
entire config block, though, that amounts to a meaningful feature.

On 08/20/2018 11:24 PM, Noah Misch wrote:
> On Sun, Aug 19, 2018 at 03:04:58PM -0400, Andrew Dunstan wrote:
>> On 08/18/2018 12:22 PM, Noah Misch wrote (offlist):
>>> On Sat, Aug 18, 2018 at 08:17:17AM -0400, Andrew Dunstan wrote:
>>>> Is there anything I can do to make upgrade easier?
>>> The show_log.pl web page could highlight a particular run's config change,
>>> including version number changes, when there is such a change compared to the
>>> previous run.  If it's clear enough that folks are unlikely to confuse an
>>> upgrade-induced failure with a PG-commit-induced failure, I could stop checking
>>> for post-upgrade buildfarm failures.
>> What I've done is add this to the database in a way that makes it easily
>> accessible to the web app, and added that info to the history page.
> I think that wastes prime screen real estate.  If show_history.pl is going to
> spend that space on something, I would choose commit hash.  But I liked it
> better with nothing there.
>
> The concept I had in mind was a "Config changed since last success" block
> adjacent to the "Files changed since last success" block in show_log.pl.  It
> might look like this:
>
> - 'config_env' => { 'CC' => 'gcc' },
> + 'config_env' => { 'CC' => 'gcc -mips32r2' },
> - 'script_version' => 'REL_7',
> + 'script_version' => 'REL_8',
>
> Script version is a minor aspect, worth little by itself.  If you diff the
> entire config block, though, that amounts to a meaningful feature.


It's easy to revert the display change, and there is utility in having 
the script version easily available in the database without having to 
extract it in each query.

Getting the config summary from the previous build is possibly a bit 
less simple.

I'll take a look when I get a bit more time.

cheers

andrew


-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Am Freitag, den 17.08.2018, 17:27 -0400 schrieb Andrew Dunstan:
>   chub          | bernd.helmle@credativ.de    | 005000

chub should be up to date now and builds REL_11_STABLE, too.

On 08/29/2018 10:35 AM, Dave Page wrote:
>
>
> On Fri, Aug 17, 2018 at 10:27 PM, Andrew Dunstan 
> <andrew.dunstan@2ndquadrant.com 
> <mailto:andrew.dunstan@2ndquadrant.com>> wrote:
>
>
>     Animal owners,
>
>
>     The majority of buildfarm animals are not on the latest client
>     code release. Version 8 was released more than 3 months ago. There
>     are 26 animals on release 7 and 43 on even older releases, in one
>     case on a release going back 4 years. See below for the lists of
>     delinquents.
>
>
>     If this situation doesn't improve I'm going to have to reinstitute
>     the server filters which will reject very old clients.
>
>
>     I do my best to keep things backwards compatible, so that usually
>     an upgrade is simply a matter of unpacking the release and you're
>     done. I don't think asking people to do that once every few months
>     is terribly onerous. It's a five minute job.
>
>
> It's very far from a 5 minute job on ancient Solaris boxes :-(
>
> In my case, I've spent an hour or more trying to get Digest::SHA to 
> install, and have failed miserably as it fails to compile for me with 
> both Sun Studio and GCC. In the case of Sun Studio, CPAN seems 
> insistent on trying to use the compiler in a way that makes it react 
> like the crippled default compiler on the system, and it ignores 
> variables like CC, and aliases and so-on, and insists on trying to run 
> "cc" regardless. I'm going to have to give up for now.
>
>


eep!

Please let me know when issues like this arise.

Note BTW that we switched from Digest::SHA1 to Digest::SHA back in 2013, 
because it's more commonly available, and I never heard of a problem 
with that until now, AFAIR. Assuming you have Digest::SHA1 available, I 
could probably save you all this grief by trying to use one and then the 
other.

More generally, if people have difficulties I try to work out a way 
around them. So please do sing out when that happens.

cheers

andrew


-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> On 08/29/2018 10:35 AM, Dave Page wrote:
>> It's very far from a 5 minute job on ancient Solaris boxes :-(

> Please let me know when issues like this arise.

One thing of the same ilk is that a few months ago I had to give up on
using https reporting on gaur/pademelon, and switch back to http.
I don't think that was your fault, because it was not associated with a
buildfarm client upgrade.  It appeared to me that the pginfra folk had
changed something to require newer TLS versions in https connections to
the buildfarm server.  Like Dave, I spent a fair while trying to install
new-enough Perl modules to fix that, without a lot of success: newer
Perl code doesn't compile on that platform.

It's debatable of course how much anyone cares about buildfarm animals
that are this old.  I've already shut down pademelon since its compiler
never heard of C99, and I don't see much point in running it on just
the back branches.

            regards, tom lane

On Thu, Aug 30, 2018 at 11:32:30AM +0100, Dave Page wrote:
> On Wed, Aug 29, 2018 at 11:17 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> 
> > Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> > > On 08/29/2018 10:35 AM, Dave Page wrote:
> > >> It's very far from a 5 minute job on ancient Solaris boxes :-(
> >
> > > Please let me know when issues like this arise.
> >
> > One thing of the same ilk is that a few months ago I had to give up on
> > using https reporting on gaur/pademelon, and switch back to http.
> > I don't think that was your fault, because it was not associated with a
> > buildfarm client upgrade.  It appeared to me that the pginfra folk had
> > changed something to require newer TLS versions in https connections to
> > the buildfarm server.  Like Dave, I spent a fair while trying to install
> > new-enough Perl modules to fix that, without a lot of success: newer
> > Perl code doesn't compile on that platform.
> >
> > It's debatable of course how much anyone cares about buildfarm animals
> > that are this old.  I've already shut down pademelon since its compiler
> > never heard of C99, and I don't see much point in running it on just
> > the back branches.
> >
> 
> Right - I do wonder that about this machine (whether it's worth supporting,
> not whether its compiler supports C99). Here's what we're talking about:
> 
> -bash-3.00$ cat /etc/release
>                        Solaris 10 11/06 s10s_u3wos_10 SPARC
>            Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
>                         Use is subject to license terms.
>                            Assembled 14 November 2006

The lack of any other Solaris animal makes it more valuable than its age alone
would imply.