BUG #13985: Segmentation fault on PREPARE TRANSACTION

The following bug has been logged on the website:

Bug reference:      13985
Logged by:          Chris Tessels
Email address:      chris.tessels@inergy.nl
PostgreSQL version: 9.4.6
Operating system:   CentOS release 6.6 (Final)
Description:

The first time we saw this segmentation fault was on an instance running
PostgreSQL 9.4.4.
To make sure the problem wasn't already fixed, we upgraded another instance
on identical hardware to PostgreSQL 9.4.6.
This would also exclude potential hardware related problems like bad
memory.
We also updated our JDBC driver to the latest version:
https://jdbc.postgresql.org/download/postgresql-9.4.1208.jar

Although we were able to make both systems crash, we were not able to
systematically reproduce the problem.

Here is the postgresql.log when the segfault occurred.

    /data/postgres/pg_log/postgresql.log
    2016-02-23 14:45:12.901 CET  LOG:  server process (PID 519) was terminated
by signal 11: Segmentation fault
    2016-02-23 14:45:12.901 CET  DETAIL:  Failed process was running: PREPARE
TRANSACTION
'131077_AAAAAAAAAAAAAP//CjIGBaCG30NWzFRpAdtPATE=_AAAAAAAAAAAAAP//CjIGBaCG30NWzFRpAdtP4AAAAAIAAAAA'
    2016-02-23 14:45:12.901 CET  LOG:  terminating any other active server
processes
    2016-02-23 14:45:12.902 CET  LOG:  archiver process (PID 114131) exited
with exit code 1
    2016-02-23 14:45:14.226 CET replication_user FATAL:  the database system is
in recovery mode
    2016-02-23 14:45:14.564 CET  LOG:  all server processes terminated;
reinitializing
    2016-02-23 14:45:17.404 CET  LOG:  database system was interrupted; last
known up at 2016-02-23 14:31:36 CET
    2016-02-23 14:53:06.753 CET mailinfo_ow FATAL:  the database system is in
recovery mode
    2016-02-23 14:53:06.760 CET mailinfo_ow FATAL:  the database system is in
recovery mode
    2016-02-23 14:53:08.830 CET replication_user FATAL:  the database system is
in recovery mode
    2016-02-23 14:53:10.203 CET  LOG:  record with zero length at 5/1409A540
    2016-02-23 14:53:10.203 CET  LOG:  redo done at 5/1409A3C0
    2016-02-23 14:53:10.203 CET  LOG:  last completed transaction was at log
time 2016-02-23 14:44:32.012791+01
    2016-02-23 14:53:11.708 CET  LOG:  MultiXact member wraparound protections
are now enabled
    2016-02-23 14:53:11.710 CET  LOG:  recovering prepared transaction
12609585
    2016-02-23 14:53:11.710 CET  LOG:  recovering prepared transaction
12609594
    2016-02-23 14:53:11.710 CET  LOG:  recovering prepared transaction
12609596
    2016-02-23 14:53:11.710 CET  LOG:  recovering prepared transaction
12609601
    2016-02-23 14:53:11.710 CET  LOG:  recovering prepared transaction
12609591
    2016-02-23 14:53:11.710 CET  LOG:  recovering prepared transaction
12609588
    2016-02-23 14:53:11.710 CET  LOG:  recovering prepared transaction
12609593
    2016-02-23 14:53:11.710 CET  LOG:  recovering prepared transaction
12609584
    2016-02-23 14:53:11.710 CET  LOG:  recovering prepared transaction
12609595
    2016-02-23 14:53:11.710 CET  LOG:  recovering prepared transaction
12609586
    2016-02-23 14:53:11.710 CET  LOG:  recovering prepared transaction
12609590
    2016-02-23 14:53:11.756 CET  LOG:  autovacuum launcher started
    2016-02-23 14:53:11.757 CET  LOG:  database system is ready to accept
connections

    /var/log/messages
    Feb 23 14:44:32 [hostname] kernel: postmaster[519]: segfault at
7fc2d8e6b634 ip 000000000066a95b sp 00007fff5ca887d0 error 4 in
postgres[400000+566000]


Analyzing the coredump gave us the following information:
    sudo -u postgres gdb -q -c
/var/spool/abrt/ccpp-2016-02-23-16\:27\:50-77566/coredump
/usr/pgsql-9.4/bin/postgres

    Core was generated by `postgres: mailinfo_ow mailinfo_ods 10.50.6.6(4188'.
    Program terminated with signal 11, Segmentation fault.

    #0  MinimumActiveBackends (min=50) at procarray.c:2472
    2472            if (pgxact->xid == InvalidTransactionId)


Postgresql version:
    PostgreSQL 9.4.6 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.4.7
20120313 (Red Hat 4.4.7-16), 64-bit


Operating system and version:
    [root@[hostname] ccpp-2016-02-23-16:27:50-77566]# cat /etc/redhat-release
    CentOS release 6.6 (Final)
    [root@[hostname] ccpp-2016-02-23-16:27:50-77566]# uname -a
    Linux [hostname] 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 04:27:16 UTC 2014
x86_64 x86_64 x86_64 GNU/Linux


We installed Postgres using the following command:
    yum install from http://yum.postgresql.org/9.4/redhat/rhel-6.6-x86_64/


Our configuration settings are:
    name    current_setting    source
    archive_command    rsync -a %p /data/postgres_archive/%f    configuration file
    archive_mode    on    configuration file
    bgwriter_delay    100ms    configuration file
    bgwriter_lru_maxpages    1000    configuration file
    checkpoint_completion_target    0    configuration file
    checkpoint_segments    192    configuration file
    checkpoint_timeout    1h    configuration file
    client_encoding    UTF8    client
    commit_delay    50    configuration file
    commit_siblings    50    configuration file
    DateStyle    ISO, MDY    client
    default_text_search_config    pg_catalog.english    configuration file
    dynamic_shared_memory_type    posix    configuration file
    effective_cache_size    16GB    configuration file
    effective_io_concurrency    1    configuration file
    extra_float_digits    3    session
    fsync    on    configuration file
    full_page_writes    off    configuration file
    hot_standby    on    configuration file
    hot_standby_feedback    on    configuration file
    lc_messages    en_US.UTF-8    configuration file
    lc_monetary    en_US.UTF-8    configuration file
    lc_numeric    en_US.UTF-8    configuration file
    lc_time    en_US.UTF-8    configuration file
    listen_addresses    *    configuration file
    log_destination    stderr    configuration file
    log_line_prefix    %m %u     configuration file
    log_min_messages    log    configuration file
    log_rotation_age    0    configuration file
    log_rotation_size    1000MB    configuration file
    log_statement    none    configuration file
    log_timezone    CET    configuration file
    log_truncate_on_rotation    on    configuration file
    logging_collector    on    configuration file
    maintenance_work_mem    2GB    configuration file
    max_connections    400    configuration file
    max_prepared_transactions    100    configuration file
    max_stack_depth    2MB    environment variable
    max_standby_archive_delay    2min    configuration file
    max_standby_streaming_delay    2min    configuration file
    max_wal_senders    3    configuration file
    port    5432    configuration file
    shared_buffers    32GB    configuration file
    synchronous_commit    off    configuration file
    temp_buffers    8MB    configuration file
    TimeZone    Europe/Amsterdam    client
    wal_level    hot_standby    configuration file
    work_mem    64MB    configuration file


The program using to connect to PostgreSQL:
    Wildfly 8.2.0 XA datasource with driver
https://jdbc.postgresql.org/download/postgresql-9.4.1208.jar

    wildfly config:

        <xa-datasource jndi-name="java:jboss/datasources/MailInfoXADS"
pool-name="MailInfoXADS" enabled="true" use-java-context="true">
            <xa-datasource-property name="ServerName">
                [hostname]
            </xa-datasource-property>
            <xa-datasource-property name="PortNumber">
                5432
            </xa-datasource-property>
            <xa-datasource-property name="DatabaseName">
                mailinfo_ods
            </xa-datasource-property>
            <driver>postgresql-jdbc4</driver>
            <xa-pool>
                <min-pool-size>5</min-pool-size>
                <initial-pool-size>5</initial-pool-size>
                <max-pool-size>40</max-pool-size>
                <prefill>true</prefill>
            </xa-pool>
            <security>
                <user-name>mailinfo_ow</user-name>
                <password>xxxxxx</password>
            </security>
            <validation>
                <valid-connection-checker
class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLValidConnectionChecker"/>
                <exception-sorter
class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLExceptionSorter"/>
            </validation>
        </xa-datasource>

        <drivers>
            <driver name="postgresql-jdbc4" module="org.postgresql">
                <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
            </driver>
        </drivers>