Обсуждение: crash of postgresql 11.2-2PGDG.rhel7 in StartupXLOG () atxlog.c:6355

Dear PostgreSQL developers,

I have automatic logging of segfaults in my lab. I have found a few
crashes of postgresql. Unfortunatelly, I am not able to dig more logs
or run any query on that machine. I believe, it may help find you some
issue.

version='postgresql 11.2-2PGDG.rhel7'
os='CentOS.7.1810'
signal='6'
pid='31494'
uid='26'
gid='26'
time='1556199867'
exe='postmaster'
exefull='!usr!pgsql-11!bin!postgres'
pid_orig='31494'
tid='31494'

2019-04-25T15:45:14+0200 Core dump saved

[New LWP 31494]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `postgres: startup   recovering
0000000100000000000000EC'.
Program terminated with signal 6, Aborted.
#0  0x00007f8c02a3e207 in raise () from /lib64/libc.so.6

Thread 1 (Thread 0x7f8c053908c0 (LWP 31494)):
#0  0x00007f8c02a3e207 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f8c02a3f8f8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x000000000084d2cb in errfinish (dummy=<optimized out>) at
elog.c:555
        edata = 0xd44560 <errordata>
        elevel = 22
        oldcontext = 0x2b713a0
        econtext = 0x0
        __func__ = "errfinish"
#3  0x000000000050580c in StartupXLOG () at xlog.c:6355
        Insert = <optimized out>
        checkPoint = <optimized out>
        wasShutdown = <optimized out>
        reachedStopPoint = <optimized out>
        haveBackupLabel = <optimized out>
        haveTblspcMap = <optimized out>
        RecPtr = <optimized out>
        checkPointLoc = <optimized out>
        EndOfLog = <optimized out>
        EndOfLogTLI = <optimized out>
        PrevTimeLineID = <optimized out>
        record = <optimized out>
        oldestActiveXID = <optimized out>
        backupEndRequired = <optimized out>
        backupFromStandby = <optimized out>
        dbstate_at_startup = <optimized out>
        xlogreader = <optimized out>
        private = {emode = 15, fetching_ckpt = true, randAccess = true}
        fast_promoted = false
        st = {st_dev = 18171478800, st_ino = 18184179600, st_nlink =
18202928400, st_mode = 1035760016, st_uid = 4, st_gid = 1054508816,
__pad0 = 4, st_rdev = 18247683600, st_size = 18265827600, st_blksize =
18279133200, st_blocks = 18297277200, st_atim = {tv_sec = 18310582800,
tv_nsec = 18328726800}, st_mtim = {tv_sec = 18342032400, tv_nsec =
18360781200}, st_ctim = {tv_sec = 18373482000, tv_nsec = 18392230800},
__unused = {18404931600, 18423680400, 18436986000}}
        __func__ = "StartupXLOG"
#4  0x00000000006d82b1 in StartupProcessMain () at startup.c:211
No locals.
#5  0x0000000000512275 in AuxiliaryProcessMain (argc=argc@entry=2, 
argv=argv@entry=0x7ffe5af831e0) at bootstrap.c:441
        progname = 0x89a46c "postgres"
        flag = <optimized out>
        userDoption = 0x0
        __func__ = "AuxiliaryProcessMain"
#6  0x00000000006d53a0 in StartChildProcess (type=StartupProcess) at
postmaster.c:5331
        pid = <optimized out>
        av = {0x89a46c "postgres", 0x7ffe5af83230 "-x2", 0x0, 0x874c61
<MemoryContextResetOnly+49> "\306C\004\001[\303f\017\037\204", 0xd45650
<context_freelists+16> "\003", 0x86edf8 <AllocSetDelete+184>
"\351p\377\377\377\017\037", 0x0, 0x2bbc8f8 "\336", 0x2bbbf50 "\325",
0x2b71300 "0\023\267\002"}
        ac = 2
        typebuf = "-
x2\000\000\000\000\000\000q\264\370\ai\350\v\003\000\000\000\000\000\00
0\000\000\023\267\002\000\000\000"
#7  0x00000000006d7b75 in PostmasterMain (argc=argc@entry=3, 
argv=argv@entry=0x2b71300) at postmaster.c:1371
        opt = <optimized out>
        status = <optimized out>
        userDoption = <optimized out>
        listen_addr_saved = true
        i = <optimized out>
        output_config_variable = <optimized out>
        __func__ = "PostmasterMain"
#8  0x000000000048124f in main (argc=3, argv=0x2b71300) at main.c:228
No locals.



regards,
-- 
Jozef Mlich <jozef.mlich@greycortex.com>

On Mon, 2019-04-29 at 10:42 +0200, Dmitry Dolgov wrote:
> > On Mon, Apr 29, 2019 at 10:16 AM Jozef Mlich <
> > jozef.mlich@greycortex.com> wrote:
> > 
> > I have automatic logging of segfaults in my lab. I have found a few
> > crashes of postgresql. Unfortunatelly, I am not able to dig more
> > logs
> > or run any query on that machine. I believe, it may help find you
> > some
> > issue.
> 
> Thanks for reporting!
> 
> > [New LWP 31494]
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > Core was generated by `postgres: startup   recovering
> > 0000000100000000000000EC'.
> > Program terminated with signal 6, Aborted.
> > #0  0x00007f8c02a3e207 in raise () from /lib64/libc.so.6
> > 
> > Thread 1 (Thread 0x7f8c053908c0 (LWP 31494)):
> > #0  0x00007f8c02a3e207 in raise () from /lib64/libc.so.6
> > No symbol table info available.
> > #1  0x00007f8c02a3f8f8 in abort () from /lib64/libc.so.6
> > No symbol table info available.
> > #2  0x000000000084d2cb in errfinish (dummy=<optimized out>) at
> > elog.c:555
> >         edata = 0xd44560 <errordata>
> >         elevel = 22
> >         oldcontext = 0x2b713a0
> >         econtext = 0x0
> >         __func__ = "errfinish"
> > #3  0x000000000050580c in StartupXLOG () at xlog.c:6355
> 
> Quick look at this version shows that it's
> 
>     ereport(FATAL, (errmsg("control file contains invalid data")));
> 
> So I guess the chances are high, that this stack trace is not an
> immediate
> result of some issue, but just a statement that something wrong
> happened to a
> control file before.

From further discussion we have identified that we stareted
'/usr/pgsql-11/bin/pg_resetwal -x' with incorrect parameter. I am not
sure if it is a bug or intended behaviour. 

-- 
Jozef Mlich <jozef.mlich@greycortex.com>

On Mon, Apr 29, 2019 at 10:49:11AM +0200, Jozef Mlich wrote:
> From further discussion we have identified that we stareted
> '/usr/pgsql-11/bin/pg_resetwal -x' with incorrect parameter. I am not
> sure if it is a bug or intended behaviour.

It is so easy to screw things up with pg_resetwal that we ought to
rename it pg_please_corrupt_my_data.
--
Michael