Обсуждение: buildfarm instance bichir stuck

On Wed, Apr 7, 2021 at 5:44 PM Robins Tharakan <tharakan@gmail.com> wrote:
> Bichir's been stuck for the past month and is unable to run regression tests since
6a2a70a02018d6362f9841cc2f499cc45405e86b.

Hrmph.  That's "Use signalfd(2) for epoll latches."  I had a similar
report from an illumos user (but it was intermittent).  I have never
seen such a failure on Linux.  My first guess is that these two
systems that are doing Linux system call emulation have implemented
subtly different semantics, and something is going wrong like this: a
SIGUSR1 arrives to tell you some important news about a procsignal and
the signal handler calls SetLatch(MyLatch) which does kill(MyProcPid,
SIGURG), but somehow that fails to wake up the epoll() you are
sleeping in which contains the signalfd that should receive the signal
and report it by being readable, due to some internal race.  Or
something like that.  But I haven't been able to verify that theory
because I don't have any of those computers.  If it is indeed
something like that and not a bug in my code, then I was thinking that
the main tool available to deal with it would be to set WAIT_USE_POLL
in the relevant template file, so that we don't use the combination of
epoll + signalfd on illlumos, but then WSL1 thows a spanner in the
works because AFAIK it's masquerading as Ubuntu, running PostgreSQL
from an Ubuntu package with a freaky kernel.  Hmm.

> It is interesting that that commit's a month old and probably no other client has complained since, but diving in, I
cansee that it's been unable to even start regression tests after that commit went in. 

Oh, well at least it's easily reproducible then, that's something!

> Note that Bichir is running on WSL1 (not WSL2) - i.e. Windows Subsystem for Linux inside Windows 10 - and so isn't
reallyproduction use-case. The only run that actually got submitted to Buildfarm was from a few days back when I killed
itafter a long wait - see [1]. 
>
> Since yesterday, I have another run that's again stuck on CREATE DATABASE (see outputs below) and although pstack not
workingmay be a limitation of the architecture / installation (unsure), a trace shows it is stuck at poll. 

That's actually the client.  I guess there is also a backend process
stuck somewhere in epoll_wait()?

On 4/7/21 2:16 AM, Thomas Munro wrote:
> On Wed, Apr 7, 2021 at 5:44 PM Robins Tharakan <tharakan@gmail.com> wrote:
>> Bichir's been stuck for the past month and is unable to run regression tests since
6a2a70a02018d6362f9841cc2f499cc45405e86b.
> Hrmph.  That's "Use signalfd(2) for epoll latches."  I had a similar
> report from an illumos user (but it was intermittent).  I have never
> seen such a failure on Linux.  My first guess is that these two
> systems that are doing Linux system call emulation have implemented
> subtly different semantics, and something is going wrong like this: a
> SIGUSR1 arrives to tell you some important news about a procsignal and
> the signal handler calls SetLatch(MyLatch) which does kill(MyProcPid,
> SIGURG), but somehow that fails to wake up the epoll() you are
> sleeping in which contains the signalfd that should receive the signal
> and report it by being readable, due to some internal race.  Or
> something like that.  But I haven't been able to verify that theory
> because I don't have any of those computers.  If it is indeed
> something like that and not a bug in my code, then I was thinking that
> the main tool available to deal with it would be to set WAIT_USE_POLL
> in the relevant template file, so that we don't use the combination of
> epoll + signalfd on illlumos, but then WSL1 thows a spanner in the
> works because AFAIK it's masquerading as Ubuntu, running PostgreSQL
> from an Ubuntu package with a freaky kernel.  Hmm.
>

To test this the OP could just add


    CPPFLAGS => '-DWAIT_USE_POLL',


to his animal's config's config_env stanza.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Robins Tharakan <tharakan@gmail.com> writes:
> Not sure if many agree but 2 things stood out here:
> 1) Buildfarm never got the message that a commit broke an instance. Ideally
> I'd have expected buildfarm to have an optimistic timeout that could have
> helped - for e.g. right now, the CREATE DATABASE is still stuck since 18
> hrs.

As far as that goes, you can set wait_timeout in the animal's config
to something comfortably more than the longest run time you expect.
It doesn't default to enabled though, possibly because picking a
one-size-fits-all value would be impossible.

I do use it on some of my flakier dinosaurs, and I've noticed that
when it does kick in, the buildfarm run just stops dead and no report
is sent to the BF server.  That has advantages in not cluttering the
BF status with run-failed-because-of-$weird_problem issues, but it
doesn't help from the standpoint of noticing when your animal is stuck.
Maybe it'd be better to change that behavior.

(I can also attest from personal experience that what had been a
comfortable amount of slop when you picked it tends to become less
so over time.  Consider yourself warned.)

            regards, tom lane

On 4/7/21 1:07 PM, Tom Lane wrote:
> Robins Tharakan <tharakan@gmail.com> writes:
>> Not sure if many agree but 2 things stood out here:
>> 1) Buildfarm never got the message that a commit broke an instance. Ideally
>> I'd have expected buildfarm to have an optimistic timeout that could have
>> helped - for e.g. right now, the CREATE DATABASE is still stuck since 18
>> hrs.
> As far as that goes, you can set wait_timeout in the animal's config
> to something comfortably more than the longest run time you expect.
> It doesn't default to enabled though, possibly because picking a
> one-size-fits-all value would be impossible.
>
> I do use it on some of my flakier dinosaurs, and I've noticed that
> when it does kick in, the buildfarm run just stops dead and no report
> is sent to the BF server.  That has advantages in not cluttering the
> BF status with run-failed-because-of-$weird_problem issues, but it
> doesn't help from the standpoint of noticing when your animal is stuck.
> Maybe it'd be better to change that behavior.
>

Yeah, I'll have a look. It's not simple for a bunch of reasons.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Andrew Dunstan <andrew@dunslane.net> writes:
> On 4/7/21 1:07 PM, Tom Lane wrote:
>> I do use it on some of my flakier dinosaurs, and I've noticed that
>> when it does kick in, the buildfarm run just stops dead and no report
>> is sent to the BF server.  That has advantages in not cluttering the
>> BF status with run-failed-because-of-$weird_problem issues, but it
>> doesn't help from the standpoint of noticing when your animal is stuck.
>> Maybe it'd be better to change that behavior.

> Yeah, I'll have a look. It's not simple for a bunch of reasons.

On further thought, that doesn't seem like the place to fix it.
I'd rather be able to ask the buildfarm server to send me nagmail
if my animal hasn't sent a report in N days (where N had better
be owner-configurable).  This would catch not only animal-is-hung,
but also other classes of problems like whole-machine-is-hung or
you-broke-your-firewall-configuration-so-it-cant-contact-the-server.
I've had issues of those sorts before ...

            regards, tom lane

On 4/7/21 4:02 PM, Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>> On 4/7/21 1:07 PM, Tom Lane wrote:
>>> I do use it on some of my flakier dinosaurs, and I've noticed that
>>> when it does kick in, the buildfarm run just stops dead and no report
>>> is sent to the BF server.  That has advantages in not cluttering the
>>> BF status with run-failed-because-of-$weird_problem issues, but it
>>> doesn't help from the standpoint of noticing when your animal is stuck.
>>> Maybe it'd be better to change that behavior.
>> Yeah, I'll have a look. It's not simple for a bunch of reasons.
> On further thought, that doesn't seem like the place to fix it.
> I'd rather be able to ask the buildfarm server to send me nagmail
> if my animal hasn't sent a report in N days (where N had better
> be owner-configurable).  This would catch not only animal-is-hung,
> but also other classes of problems like whole-machine-is-hung or
> you-broke-your-firewall-configuration-so-it-cant-contact-the-server.
> I've had issues of those sorts before ...
>
>             



That already exists, and has for a long time. See the 'alerts' stanza of
your config file.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Andrew Dunstan <andrew@dunslane.net> writes:
> On 4/7/21 4:02 PM, Tom Lane wrote:
>> On further thought, that doesn't seem like the place to fix it.
>> I'd rather be able to ask the buildfarm server to send me nagmail
>> if my animal hasn't sent a report in N days (where N had better
>> be owner-configurable).

> That already exists, and has for a long time. See the 'alerts' stanza of
> your config file.

Oh!  In that case, I don't think we need anything else.

            regards, tom lane

On Wed, Apr 7, 2021 at 7:31 PM Robins Tharakan <tharakan@gmail.com> wrote:
> Correct. This is easily reproducible on this test-instance, so let me know if you want me to test a patch.

From your description it sounds like signals are not arriving at all,
rather than some more complicated race.  Let's go back to basics...
what does the attached program print for you?  I see:

tmunro@x1:~/junk$ cc test-signalfd.c
tmunro@x1:~/junk$ ./a.out
blocking SIGURG...
creating a signalfd to receive SIGURG...
creating an epoll set...
adding signalfd to epoll set...
polling the epoll set... 0
sending a signal...
polling the epoll set... 1

On Fri, Apr 9, 2021 at 6:11 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Wed, Apr 7, 2021 at 7:31 PM Robins Tharakan <tharakan@gmail.com> wrote:
> > Correct. This is easily reproducible on this test-instance, so let me know if you want me to test a patch.
>
> From your description it sounds like signals are not arriving at all,
> rather than some more complicated race.  Let's go back to basics...

I was looking into the portability of SIGURG and OOB socket data for
something totally different (hallway track discussion from PGCon,
could we use that for query cancel, like FTP does, instead of opening
another socket?), and lo and behold, someone has figured out a
workaround for this latch problem:

https://github.com/microsoft/WSL/issues/8619

I don't really want to add code to scrape uname() ouput detect
different kernels at runtime as shown there, but it doesn't seem to
make a difference on Linux if we just always do what was suggested.  I
didn't look too hard into whether that is the right place to put the
call, or really understand *why* it works, and since I am not a
Windows user and we don't have a WSL1 CI, I can't confirm that it
works or explore whether there is some other ordering of operations
that would be better but still work, but if that does the trick then
maybe we should just do something like the attached.

Thoughts?