The OOM was at:
< 2022-09-09 19:34:24.043 CDT >LOG: server process (PID 14841) was terminated by signal 9: Killed
The first SIGABRT was at:
< 2022-09-09 19:37:31.650 CDT >LOG: server process (PID 7363) was terminated by signal 6: Aborted
And I've just found a bunch of "interesting" logs between the two:
< 2022-09-09 19:36:48.505 CDT telsasoft >ERROR: MultiXactId 133553150 has not been created yet -- apparent wraparound
< 2022-09-09 19:36:48.505 CDT telsasoft >STATEMENT: SELECT alarm_id, alarm_disregard FROM alarms WHERE alarm_ack_time
ISNULL AND alarm_clear_time IS NULL AND alarm_office = 'ETHERNET'
< 2022-09-09 19:36:48.788 CDT telsasoft >ERROR: could not access status of transaction 3779944583
< 2022-09-09 19:36:48.788 CDT telsasoft >DETAIL: Could not read from file "pg_subtrans/E14D" at offset 98304: read too
fewbytes.
...
< 2022-09-09 19:37:08.550 CDT telsasoft >ERROR: MultiXactId 133553156 has not been created yet -- apparent wraparound
...
< 2022-09-09 19:37:13.792 CDT telsasoft >ERROR: could not access status of transaction 3779946306
< 2022-09-09 19:37:13.792 CDT telsasoft >DETAIL: Could not read from file "pg_subtrans/E14D" at offset 98304: read too
fewbytes.
...
< 2022-09-09 19:37:19.682 CDT telsasoft >ERROR: could not access status of transaction 3779946306
< 2022-09-09 19:37:19.682 CDT telsasoft >DETAIL: Could not read from file "pg_subtrans/E14D" at offset 98304: read too
fewbytes.
< 2022-09-09 19:37:19.682 CDT telsasoft >CONTEXT: while locking tuple (11755,5) in relation "alarms_null"
...
< 2022-09-09 19:37:25.835 CDT telsasoft >ERROR: MultiXactId 133553154 has not been created yet -- apparent wraparound
BTW, if I load the datadir backup to crash it, I see:
t_infomask = 4423, which is:
; 0x1000+0x0100+0x0040+0x0004+0x0002+0x0001
4423
src/include/access/htup_details.h-#define HEAP_HASNULL 0x0001 /* has null attribute(s) */
src/include/access/htup_details.h-#define HEAP_HASVARWIDTH 0x0002 /* has variable-width attribute(s) */
src/include/access/htup_details.h-#define HEAP_HASEXTERNAL 0x0004 /* has external stored attribute(s) */
src/include/access/htup_details.h-#define HEAP_XMAX_EXCL_LOCK 0x0040 /* xmax is exclusive locker */
src/include/access/htup_details.h-#define HEAP_XMIN_COMMITTED 0x0100 /* t_xmin committed */
src/include/access/htup_details.h:#define HEAP_XMAX_IS_MULTI 0x1000 /* t_xmax is a MultiXactId */
I was I could say what autovacuum had been doing during that period, but
unfortunately I have "log_autovacuum_min_duration = 9s"...