Re: Why is infinite_recurse test suddenly failing?

Thomas Munro <thomas.munro@gmail.com> writes:
> Here's another crash like that.

> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=cavefish&dt=2019-07-13%2003%3A49%3A38
> 2019-07-13 04:01:23.437 UTC [9365:70] LOG:  server process (PID 12951)
> was terminated by signal 11: Segmentation fault
> 2019-07-13 04:01:23.437 UTC [9365:71] DETAIL:  Failed process was
> running: select infinite_recurse();

It occurred to me to scrape the buildfarm database for these failures,
and what I got was

   sysname    |    branch     |      snapshot       |      stage      |
   data                                                      |   architecture 

--------------+---------------+---------------------+-----------------+---------------------------------------------------------------------------------------------------------------+------------------
 demoiselle   | HEAD          | 2019-04-27 14:55:52 | pg_upgradeCheck | 2019-04-27 15:00:42.736 UTC [1457:66] DETAIL:
Failedprocess was running: select infinite_recurse();         | ppc64le (POWER9)  
 buri         | HEAD          | 2019-04-27 23:54:46 | Check           | 2019-04-28 00:01:49.794 UTC [3041:66] DETAIL:
Failedprocess was running: select infinite_recurse();         | ppc64le (POWER9)  
 takin        | HEAD          | 2019-05-01 08:16:48 | pg_upgradeCheck | 2019-05-01 08:23:27.159 UTC [32303:59] DETAIL:
Failedprocess was running: select infinite_recurse();        | ppc64le           
 bonito       | HEAD          | 2019-05-01 23:05:36 | Check           | 2019-05-01 23:11:00.145 UTC [13933:66] DETAIL:
Failedprocess was running: select infinite_recurse();        | ppc64le (POWER9)  
 shoveler     | HEAD          | 2019-05-10 14:04:34 | Check           | 2019-05-10 14:11:26.833 UTC [13456:73] DETAIL:
Failedprocess was running: select infinite_recurse();        | ppc64le (POWER8)  
 demoiselle   | HEAD          | 2019-05-19 14:22:23 | pg_upgradeCheck | 2019-05-19 14:26:17.002 UTC [23275:80] DETAIL:
Failedprocess was running: select infinite_recurse();        | ppc64le (POWER9)  
 vulpes       | HEAD          | 2019-06-15 09:16:45 | pg_upgradeCheck | 2019-06-15 09:22:22.268 UTC [4885:77] DETAIL:
Failedprocess was running: select infinite_recurse();         | ppc64le           
 ayu          | HEAD          | 2019-06-19 22:13:23 | pg_upgradeCheck | 2019-06-19 22:18:16.805 UTC [2708:71] DETAIL:
Failedprocess was running: select infinite_recurse();         | ppc64le (POWER8)  
 quokka       | HEAD          | 2019-07-10 14:20:13 | pg_upgradeCheck | 2019-07-10 15:24:06.102 BST [5d25f4fb.2644:5]
DETAIL: Failed process was running: select infinite_recurse(); | ppc64             
 cavefish     | HEAD          | 2019-07-13 03:49:38 | pg_upgradeCheck | 2019-07-13 04:01:23.437 UTC [9365:71] DETAIL:
Failedprocess was running: select infinite_recurse();         | ppc64le (POWER9)  
 pintail      | REL_12_STABLE | 2019-07-13 19:36:51 | Check           | 2019-07-13 19:39:29.013 UTC [31086:5] DETAIL:
Failedprocess was running: select infinite_recurse();         | ppc64le (POWER9)  
 bonito       | HEAD          | 2019-07-19 23:13:01 | Check           | 2019-07-19 23:16:33.330 UTC [24191:70] DETAIL:
Failedprocess was running: select infinite_recurse();        | ppc64le (POWER9)  
 takin        | HEAD          | 2019-07-24 08:24:56 | Check           | 2019-07-24 08:28:01.735 UTC [16366:75] DETAIL:
Failedprocess was running: select infinite_recurse();        | ppc64le           
 quokka       | HEAD          | 2019-07-31 02:00:07 | pg_upgradeCheck | 2019-07-31 03:04:04.043 BST [5d40f709.776a:5]
DETAIL: Failed process was running: select infinite_recurse(); | ppc64             
 elasmobranch | HEAD          | 2019-08-01 03:13:38 | Check           | 2019-08-01 03:19:05.394 UTC [22888:62] DETAIL:
Failedprocess was running: select infinite_recurse();        | ppc64le (POWER9)  
 buri         | HEAD          | 2019-08-02 00:10:23 | Check           | 2019-08-02 00:17:11.075 UTC [28222:73] DETAIL:
Failedprocess was running: select infinite_recurse();        | ppc64le (POWER9)  
 urocryon     | HEAD          | 2019-08-02 05:43:46 | Check           | 2019-08-02 05:51:51.944 UTC [2724:64] DETAIL:
Failedprocess was running: select infinite_recurse();         | ppc64le           
 batfish      | HEAD          | 2019-08-04 19:02:36 | pg_upgradeCheck | 2019-08-04 19:08:11.728 UTC [23899:79] DETAIL:
Failedprocess was running: select infinite_recurse();        | ppc64le           
 buri         | REL_12_STABLE | 2019-08-07 00:03:29 | pg_upgradeCheck | 2019-08-07 00:11:24.500 UTC [1405:5] DETAIL:
Failedprocess was running: select infinite_recurse();          | ppc64le (POWER9)  
 quokka       | REL_12_STABLE | 2019-08-08 02:43:45 | pg_upgradeCheck | 2019-08-08 03:47:38.115 BST [5d4b8d3f.cdd7:5]
DETAIL: Failed process was running: select infinite_recurse(); | ppc64             
 quokka       | HEAD          | 2019-08-08 14:00:08 | Check           | 2019-08-08 15:02:59.770 BST [5d4c2b88.cad9:5]
DETAIL: Failed process was running: select infinite_recurse(); | ppc64             
 mereswine    | REL_11_STABLE | 2019-08-11 02:10:12 | InstallCheck-C  | 2019-08-11 02:36:10.159 PDT [5004:4] DETAIL:
Failedprocess was running: select infinite_recurse();          | ARMv7             
 takin        | HEAD          | 2019-08-11 08:02:48 | Check           | 2019-08-11 08:05:57.789 UTC [11500:67] DETAIL:
Failedprocess was running: select infinite_recurse();        | ppc64le           
 mereswine    | REL_12_STABLE | 2019-08-11 09:52:46 | pg_upgradeCheck | 2019-08-11 04:21:16.756 PDT [6804:5] DETAIL:
Failedprocess was running: select infinite_recurse();          | ARMv7             
 mereswine    | HEAD          | 2019-08-11 11:29:27 | pg_upgradeCheck | 2019-08-11 07:15:28.454 PDT [9954:76] DETAIL:
Failedprocess was running: select infinite_recurse();         | ARMv7             
 demoiselle   | HEAD          | 2019-08-11 14:51:38 | pg_upgradeCheck | 2019-08-11 14:57:29.422 UTC [9436:70] DETAIL:
Failedprocess was running: select infinite_recurse();         | ppc64le (POWER9)  
(26 rows)

This is from a scan going back 9 months (to mid-December), so the lack of
any matches before late April is pretty notable: it seems highly probable
that some change we made during April is related.

A cursory scan of commits during April finds only one that seems
conceivably related (though perhaps I just lack enough imagination):

Author: Tom Lane <tgl@sss.pgh.pa.us>
Branch: master Release: REL_12_BR [798070ec0] 2019-04-11 18:16:50 -0400

    Re-order some regression test scripts for more parallelism.

    Move the strings, numerology, insert, insert_conflict, select and
    errors tests to be parts of nearby parallel groups, instead of
    executing by themselves.

So that leads to the thought that "the infinite_recurse test is fine
if it runs by itself, but it tends to fall over if there are
concurrently-running backends".  I have absolutely no idea how that
would happen on anything that passes for a platform built in this
century.  Still, it's a place to start, which we hadn't before.

Also notable is that we now have a couple of hits on ARM, not
only ppc64.  Don't know what to make of that.

            regards, tom lane