Re: Recent SIGSEGV failures in buildfarm HEAD

Tom Lane wrote:
> Alvaro Herrera <alvherre@commandprompt.com> writes:
>
>> Andrew Dunstan wrote:
>>
>>> here's a quick untested patch for buildfarm that Stefan might like to try.
>>>
>
>
>> Note that not all core files are named "core".  On some Linux distros,
>> it's configured to be "core.PID" by default.
>>
>
> And on some platforms, cores don't drop in the current working directory
> ... but until we have a problem that *only* manifests on such a
> platform, I wouldn't worry about that.  We do need to look for 'core*'
> not just 'core', though.
>

That part is easy enough. And if people mangle their core location I am
certainly not going to go looking for it.
> Don't forget the ulimit point either ... on most Linuxen there won't be
> any core at all without twiddling ulimit.
>
>
>

Yeah. Perl actually doesn't have a core call for this. I have built some
code (see attached revised patch) to try to do it using a widespread but
non-standard module called BSD::Resource, but if the module is missing
it won't fail.

I'm actually wondering if unlimiting core might not be a useful switch
to provide on pg_ctl, as long as the platform has setrlimit().

cheers

andrew

--- run_build.pl.orig    2006-12-28 17:32:14.000000000 -0500
+++ run_build.pl.new    2006-12-29 10:59:39.000000000 -0500
@@ -299,6 +299,20 @@
     unlink $forcefile;
 }

+# try to allow core files to be produced.
+# another way would be for the calling environment
+# to call ulimit. We do this in an eval so failure is
+# not fatal.
+eval
+{
+    require BSD::Resource;
+    BSD::Resource->import();
+    # explicit sub calls here using & keeps compiler happy
+    my $coreok = setrlimit(&RLIMIT_CORE,&RLIM_INFINITY,&RLIM_INFINITY);
+    die "setrlimit" unless $coreok;
+};
+warn "failed to unlimit core size: $@" if $@;
+
 # the time we take the snapshot
 my $now=time;
 my $installdir = "$buildroot/$branch/inst";
@@ -795,6 +809,34 @@
     $dbstarted=undef;
 }

+
+sub get_stack_trace
+{
+    my $bindir = shift;
+    my $pgdata = shift;
+
+    # no core = no result
+    my @cores = glob("$pgdata/core*");
+    return () unless @cores;
+
+    # no gdb = no result
+    system "gdb --version > /dev/null 2>&1";
+    my $status = $? >>8;
+    return () if $status;
+
+    my @trace;
+
+    foreach my $core (@cores)
+    {
+        my @onetrace = `gdb -ex bt --batch $bindir/postgres $core 2>&1`;
+        push(@trace,
+            "\n\n================== stack trace: $core ==================\n",
+             @onetrace);
+    }
+
+    return @trace;
+}
+
 sub make_install_check
 {
     my @checkout = `cd $pgsql/src/test/regress && $make installcheck 2>&1`;
@@ -814,6 +856,11 @@
         }
         close($handle);
     }
+    if ($status)
+    {
+        my @trace = get_stack_trace("$installdir/bin","$installdir/data");
+        push(@checkout,@trace);
+    }
     writelog('install-check',\@checkout);
     print "======== make installcheck log ===========\n",@checkout
         if ($verbose > 1);
@@ -839,6 +886,11 @@
         }
         close($handle);
     }
+    if ($status)
+    {
+        my @trace = get_stack_trace("$installdir/bin","$installdir/data");
+        push(@checkout,@trace);
+    }
     writelog('contrib-install-check',\@checkout);
     print "======== make contrib installcheck log ===========\n",@checkout
         if ($verbose > 1);
@@ -864,6 +916,11 @@
         }
         close($handle);
     }
+    if ($status)
+    {
+        my @trace = get_stack_trace("$installdir/bin","$installdir/data");
+        push(@checkout,@trace);
+    }
     writelog('pl-install-check',\@checkout);
     print "======== make pl installcheck log ===========\n",@checkout
         if ($verbose > 1);
@@ -892,6 +949,13 @@
         }
         close($handle);
     }
+    if ($status)
+    {
+        my @trace =
+            get_stack_trace("$pgsql/src/test/regress/install$installdir/bin",
+                            "$pgsql/src/test/regress/tmp_check/data");
+        push(@makeout,@trace);
+    }
     writelog('check',\@makeout);
     print "======== make check logs ===========\n",@makeout
         if ($verbose > 1);