Обсуждение: Runtime error: could not open segment 1 of relation ...

This morning one of our production servers (8.1.1, Linux) threw a number of  errors like these (different final number;
firsttwo [1663/43801] are all the same]: 

2006-09-26 10:02:15.124 PDT 1387333718 ERROR:  could not open segment 1 of relation 1663/43801/7274801 (target block
478216192):No such file or directory 
2006-09-26 10:02:52.991 PDT 1387340456 ERROR:  could not open segment 1 of relation 1663/43801/7274801 (target block
478216192):No such file or directory 
2006-09-26 10:02:53.002 PDT 1387340458 ERROR:  could not open segment 1 of relation 1663/43801/7274801 (target block
478216192):No such file or directory 

The server never really recovered -- it kept running but was horribly slow. Rebooting seems to have cured the problem,
butI am wondering what I made look into, etc. 

The server in question is really a read-only beast with several hundred connections, typically, 4 gigs of RAM, and has
notmisbehaved before. 

Any suggestions would be welcome! (Although I fear I know the answer -- disk issues.)

Thanks,

Greg Williamson
DBA
GlobeXplorer LLC

Thanks for the suggestion ...no evidence of such a shutdown. The first of these errors (since 2005-10-28) was at
2006-09-2605:37:19.157 PDT today; machine was last shut serveral weeks ago. There's been several large updates since
then(sort of batch process for updates, then the server is read-only for 99.999 of the time). 

Greg

-----Original Message-----
From:    Talha Khan [mailto:talha.amjad@gmail.com]
Sent:    Tue 9/26/2006 3:21 PM
To:    Gregory S. Williamson
Cc:    pgsql-admin@postgresql.org
Subject:    Re: [ADMIN] Runtime error: could not open segment 1 of relation ...

Hi Greg,

Is it possible that the server was shutdown using -mi switch?

cuz immediate shutdown ends causing these problems.

Regards
Talha Khan


On 9/26/06, Gregory S. Williamson <gsw@globexplorer.com> wrote:
>
> This morning one of our production servers (8.1.1, Linux) threw a number
> of  errors like these (different final number; first two [1663/43801] are
> all the same]:
>
> 2006-09-26 10:02:15.124 PDT 1387333718 ERROR:  could not open segment 1 of
> relation 1663/43801/7274801 (target block 478216192): No such file or
> directory
> 2006-09-26 10:02:52.991 PDT 1387340456 ERROR:  could not open segment 1 of
> relation 1663/43801/7274801 (target block 478216192): No such file or
> directory
> 2006-09-26 10:02:53.002 PDT 1387340458 ERROR:  could not open segment 1 of
> relation 1663/43801/7274801 (target block 478216192): No such file or
> directory
>
> The server never really recovered -- it kept running but was horribly
> slow. Rebooting seems to have cured the problem, but I am wondering what I
> made look into, etc.
>
> The server in question is really a read-only beast with several hundred
> connections, typically, 4 gigs of RAM, and has not misbehaved before.
>
> Any suggestions would be welcome! (Although I fear I know the answer --
> disk issues.)
>
> Thanks,
>
> Greg Williamson
> DBA
> GlobeXplorer LLC
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>



-------------------------------------------------------
Click link below if it is SPAM gsw@globexplorer.com

"https://mailscanner.globexplorer.com/dspam/dspam.cgi?signatureID=4519a6f898846491211187&user=gsw@globexplorer.com&retrain=spam&template=history&history_page=1"
!DSPAM:4519a6f898846491211187!
-------------------------------------------------------

"Gregory S. Williamson" <gsw@globexplorer.com> writes:
>> 2006-09-26 10:02:15.124 PDT 1387333718 ERROR:  could not open segment 1 of
>> relation 1663/43801/7274801 (target block 478216192): No such file or
>> directory

This is most likely caused by trying to follow a bogus item pointer in
an index.  If the problem went away after a restart, the bogus value
could not have been on disk but must have been in a bad in-memory copy
of an index page.  Which suggests a hardware glitch ... although if you
are using contrib/ltree, I recall seeing several bug fixes go by
recently that were described as fixing potential ltree-index corruption
issues.

            regards, tom lane

Tom,

I haven't used the ltree module ... I'll take a look at our DB and see. It's possible that this was an index problem as
therewas a bulk update with indexes in tact, so there would have been a lot of index shuffling, and the db was not shut
downsince that update. 

Thanks,

Greg


-----Original Message-----
From:    Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent:    Tue 9/26/2006 3:36 PM
To:    Gregory S. Williamson
Cc:    Talha Khan; pgsql-admin@postgresql.org
Subject:    Re: [ADMIN] Runtime error: could not open segment 1 of relation ...

"Gregory S. Williamson" <gsw@globexplorer.com> writes:
>> 2006-09-26 10:02:15.124 PDT 1387333718 ERROR:  could not open segment 1 of
>> relation 1663/43801/7274801 (target block 478216192): No such file or
>> directory

This is most likely caused by trying to follow a bogus item pointer in
an index.  If the problem went away after a restart, the bogus value
could not have been on disk but must have been in a bad in-memory copy
of an index page.  Which suggests a hardware glitch ... although if you
are using contrib/ltree, I recall seeing several bug fixes go by
recently that were described as fixing potential ltree-index corruption
issues.

            regards, tom lane


-------------------------------------------------------
Click link below if it is SPAM gsw@globexplorer.com

"https://mailscanner.globexplorer.com/dspam/dspam.cgi?signatureID=4519aa77100151804284693&user=gsw@globexplorer.com&retrain=spam&template=history&history_page=1"
!DSPAM:4519aa77100151804284693!
-------------------------------------------------------

Talha,

Seems like a plan ... once we have it out of runtime I'll be able to redo indexes, whatever.

Thanks,

Greg

-----Original Message-----
From:    Talha Khan [mailto:talha.amjad@gmail.com]
Sent:    Tue 9/26/2006 3:33 PM
To:    Gregory S. Williamson
Cc:    pgsql-admin@postgresql.org
Subject:    Re: [ADMIN] Runtime error: could not open segment 1 of relation ...

Try reindexing your database and see if that solves your problem.

Regards
Talha Khan


On 9/26/06, Gregory S. Williamson <gsw@globexplorer.com> wrote:
>
> Thanks for the suggestion ...no evidence of such a shutdown. The first of
> these errors (since 2005-10-28) was at 2006-09-26 05:37:19.157 PDT today;
> machine was last shut serveral weeks ago. There's been several large updates
> since then (sort of batch process for updates, then the server is read-only
> for 99.999 of the time).
>
> Greg
>
> -----Original Message-----
> From:   Talha Khan [mailto:talha.amjad@gmail.com]
> Sent:   Tue 9/26/2006 3:21 PM
> To:     Gregory S. Williamson
> Cc:     pgsql-admin@postgresql.org
> Subject:        Re: [ADMIN] Runtime error: could not open segment 1 of
> relation ...
>
> Hi Greg,
>
> Is it possible that the server was shutdown using -mi switch?
>
> cuz immediate shutdown ends causing these problems.
>
> Regards
> Talha Khan
>
>
> On 9/26/06, Gregory S. Williamson <gsw@globexplorer.com> wrote:
> >
> > This morning one of our production servers (8.1.1, Linux) threw a number
> > of  errors like these (different final number; first two [1663/43801]
> are
> > all the same]:
> >
> > 2006-09-26 10:02:15.124 PDT 1387333718 ERROR:  could not open segment 1
> of
> > relation 1663/43801/7274801 (target block 478216192): No such file or
> > directory
> > 2006-09-26 10:02:52.991 PDT 1387340456 ERROR:  could not open segment 1
> of
> > relation 1663/43801/7274801 (target block 478216192): No such file or
> > directory
> > 2006-09-26 10:02:53.002 PDT 1387340458 ERROR:  could not open segment 1
> of
> > relation 1663/43801/7274801 (target block 478216192): No such file or
> > directory
> >
> > The server never really recovered -- it kept running but was horribly
> > slow. Rebooting seems to have cured the problem, but I am wondering what
> I
> > made look into, etc.
> >
> > The server in question is really a read-only beast with several hundred
> > connections, typically, 4 gigs of RAM, and has not misbehaved before.
> >
> > Any suggestions would be welcome! (Although I fear I know the answer --
> > disk issues.)
> >
> > Thanks,
> >
> > Greg Williamson
> > DBA
> > GlobeXplorer LLC
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 2: Don't 'kill -9' the postmaster
> >
>
>
>
> -------------------------------------------------------
> Click link below if it is SPAM gsw@globexplorer.com
> "
>
https://mailscanner.globexplorer.com/dspam/dspam.cgi?signatureID=4519a6f898846491211187&user=gsw@globexplorer.com&retrain=spam&template=history&history_page=1
> "
>
> -------------------------------------------------------
>
>
>
>



-------------------------------------------------------
Click link below if it is SPAM gsw@globexplorer.com

"https://mailscanner.globexplorer.com/dspam/dspam.cgi?signatureID=4519a9a399831804284693&user=gsw@globexplorer.com&retrain=spam&template=history&history_page=1"
!DSPAM:4519a9a399831804284693!
-------------------------------------------------------