Обсуждение: Postgres process terminating

Hi all, I'm currently running postgres-7.4.12 with slony-1.0.5 and am seeing the below postgres errors:

Mar 14 00:00:25 master postgres[27357]: [12-1] FATAL:  57P03: the database system is starting up
Mar 14 00:00:25 master postgres[27357]: [12-2] LOCATION:  ProcessStartupPacket, postmaster.c:1437
Mar 14 00:00:25 master postgres[27356]: [17-1] LOG:  00000: unexpected pageaddr 1/55F4A000 in log file 1, segment 183, offset 16031744
Mar 14 00:00:25 master postgres[27356]: [17-2] LOCATION:  ValidXLOGHeader, xlog.c:2096
Mar 14 00:00:25 master postgres[27356]: [18-1] LOG:  00000: redo done at 1/B7F49220
Mar 14 00:00:25 master postgres[27356]: [18-2] LOCATION:  StartupXLOG, xlog.c:2796
Mar 14 00:00:26 master postgres[27386]: [11-1] LOG:  00000: connection received: host=172.20.42.19 port=54294
Mar 14 00:00:26 master postgres[27386]: [11-2] LOCATION:  BackendFork, postmaster.c:2395
Mar 14 00:00:26 master postgres[27386]: [12-1] FATAL:  57P03: the database system is starting up
Mar 14 00:00:26 master postgres[27386]: [12-2] LOCATION:  ProcessStartupPacket, postmaster.c:1437
Mar 14 00:00:26 master postgres[27387]: [11-1] LOG:  00000: connection received: host=172.20.42.19 port=54295
Mar 14 00:00:26 master postgres[27387]: [11-2] LOCATION:  BackendFork, postmaster.c:2395
Mar 14 00:00:26 master postgres[27387]: [12-1] FATAL:  57P03: the database system is starting up
Mar 14 00:00:26 master postgres[27387]: [12-2] LOCATION:  ProcessStartupPacket, postmaster.c:1437
Mar 14 00:00:26 master postgres[27388]: [11-1] LOG:  00000: connection received: host=172.20.42.20 port=38111
Mar 14 00:00:26 master postgres[27388]: [11-2] LOCATION:  BackendFork, postmaster.c:2395
Mar 14 00:00:26 master postgres[27388]: [12-1] FATAL:  57P03: the database system is starting up
Mar 14 00:00:26 master postgres[27388]: [12-2] LOCATION:  ProcessStartupPacket, postmaster.c:1437
Mar 14 00:00:26 master postgres[27389]: [11-1] LOG:  00000: connection received: host=172.20.42.20 port=38112
Mar 14 00:00:26 master postgres[27389]: [11-2] LOCATION:  BackendFork, postmaster.c:2395
Mar 14 00:00:26 master postgres[27389]: [12-1] FATAL:  57P03: the database system is starting up
Mar 14 00:00:26 master postgres[27389]: [12-2] LOCATION:  ProcessStartupPacket, postmaster.c:1437
Mar 14 00:00:26 master postgres[27392]: [11-1] LOG:  00000: connection received: host=172.20.42.19 port=54297
Mar 14 00:00:26 master postgres[27392]: [11-2] LOCATION:  BackendFork, postmaster.c:2395
Mar 14 00:00:26 master postgres[27392]: [12-1] FATAL:  57P03: the database system is starting up
Mar 14 00:00:26 master postgres[27392]: [12-2] LOCATION:  ProcessStartupPacket, postmaster.c:1437
Mar 14 00:00:26 master postgres[27393]: [11-1] LOG:  00000: connection received: host=172.20.42.19 port=54298
Mar 14 00:00:26 master postgres[27393]: [11-2] LOCATION:  BackendFork, postmaster.c:2395
Mar 14 00:00:26 master postgres[27393]: [12-1] FATAL:  57P03: the database system is starting up

This causes slony to die on all clusters as well as our web applications.

In google'ing for "FATAL:  57P03: the database system is starting up" I noticed other people having similar issues but I not been able to locate a fix.

This morning I dropped all slony schema's  from the database and am continuing to re-create my clusters one by one and have not seeing these errors thus far.

What I am interested in is a description of the above error log.

Any help is appreciated! 

Thanks.


--
Cory Vokey

Hi Cory,

Actually it seems that the database wasn't shut down properly, and
while it was doing the recovery, it refused connections. Expected and
what needs to happen, indeed.

That's why if you are a client about to connect, you'll recieve the "
FATAL ...database is starting up " message.

Ideally, you (the dba) should let the server end this task (the
recovery). AFAIK, the recovery started as a consecuence of dirty data
somewere, probably a big transaction not comitted when the server went
off.

Are you aware of a shutdown prior to this messages?

Nothing to do with your current issue, but you will receive comments
related with your pgsql version if you are not using a  pgsql 8 :)

Regards
Guido Barosio.
( Shall I confess that I still have some 7.4.2 instances running in production)

On 3/14/06, Cory Vokey <cvokey@gmail.com> wrote:
> Hi all, I'm currently running postgres-7.4.12 with slony-1.0.5 and am seeing
> the below postgres errors:
>
>  Mar 14 00:00:25 master postgres[27357]: [12-1] FATAL:  57P03: the database
> system is starting up
>  Mar 14 00:00:25 master postgres[27357]: [12-2] LOCATION:
> ProcessStartupPacket, postmaster.c:1437
>  Mar 14 00:00:25 master postgres[27356]: [17-1] LOG:  00000: unexpected
> pageaddr 1/55F4A000 in log file 1, segment 183, offset 16031744
>  Mar 14 00:00:25 master postgres[27356]: [17-2] LOCATION:  ValidXLOGHeader,
> xlog.c:2096
>  Mar 14 00:00:25 master postgres[27356]: [18-1] LOG:  00000: redo done at
> 1/B7F49220
>  Mar 14 00:00:25 master postgres[27356]: [18-2] LOCATION:  StartupXLOG,
> xlog.c:2796
>  Mar 14 00:00:26 master postgres[27386]: [11-1] LOG:  00000: connection
> received: host=172.20.42.19 port=54294
>  Mar 14 00:00:26 master postgres[27386]: [11-2] LOCATION:  BackendFork,
> postmaster.c:2395
>  Mar 14 00:00:26 master postgres[27386]: [12-1] FATAL:  57P03: the database
> system is starting up
>  Mar 14 00:00:26 master postgres[27386]: [12-2] LOCATION:
> ProcessStartupPacket, postmaster.c:1437
>  Mar 14 00:00:26 master postgres[27387]: [11-1] LOG:  00000: connection
> received: host=172.20.42.19 port=54295
>  Mar 14 00:00:26 master postgres[27387]: [11-2] LOCATION:  BackendFork,
> postmaster.c:2395
>  Mar 14 00:00:26 master postgres[27387]: [12-1] FATAL:  57P03: the database
> system is starting up
>  Mar 14 00:00:26 master postgres[27387]: [12-2] LOCATION:
> ProcessStartupPacket, postmaster.c:1437
>  Mar 14 00:00:26 master postgres[27388]: [11-1] LOG:  00000: connection
> received: host=172.20.42.20 port=38111
>  Mar 14 00:00:26 master postgres[27388]: [11-2] LOCATION:  BackendFork,
> postmaster.c:2395
>  Mar 14 00:00:26 master postgres[27388]: [12-1] FATAL:  57P03: the database
> system is starting up
>  Mar 14 00:00:26 master postgres[27388]: [12-2] LOCATION:
> ProcessStartupPacket, postmaster.c:1437
>  Mar 14 00:00:26 master postgres[27389]: [11-1] LOG:  00000: connection
> received: host=172.20.42.20 port=38112
>  Mar 14 00:00:26 master postgres[27389]: [11-2] LOCATION:  BackendFork,
> postmaster.c:2395
>  Mar 14 00:00:26 master postgres[27389]: [12-1] FATAL:  57P03: the database
> system is starting up
>  Mar 14 00:00:26 master postgres[27389]: [12-2] LOCATION:
> ProcessStartupPacket, postmaster.c:1437
>  Mar 14 00:00:26 master postgres[27392]: [11-1] LOG:  00000: connection
> received: host=172.20.42.19 port=54297
>  Mar 14 00:00:26 master postgres[27392]: [11-2] LOCATION:  BackendFork,
> postmaster.c:2395
>  Mar 14 00:00:26 master postgres[27392]: [12-1] FATAL:  57P03: the database
> system is starting up
>  Mar 14 00:00:26 master postgres[27392]: [12-2] LOCATION:
> ProcessStartupPacket, postmaster.c:1437
>  Mar 14 00:00:26 master postgres[27393]: [11-1] LOG:  00000: connection
> received: host=172.20.42.19 port=54298
>  Mar 14 00:00:26 master postgres[27393]: [11-2] LOCATION:  BackendFork,
> postmaster.c:2395
>  Mar 14 00:00:26 master postgres[27393]: [12-1] FATAL:  57P03: the database
> system is starting up
>
>  This causes slony to die on all clusters as well as our web applications.
>
>  In google'ing for "FATAL:  57P03: the database system is starting up" I
> noticed other people having similar issues but I not been able to locate a
> fix.
>
>  This morning I dropped all slony schema's  from the database and am
> continuing to re-create my clusters one by one and have not seeing these
> errors thus far.
>
>  What I am interested in is a description of the above error log.
>
>  Any help is appreciated!
>
>  Thanks.
>
>
> --
> Cory Vokey


--
/"\   ASCII Ribbon Campaign  .
\ / - NO HTML/RTF in e-mail  .
 X  - NO Word docs in e-mail .
/ \ -----------------------------------------------------------------

On Tue, 2006-03-14 at 14:05, Guido Barosio wrote:
> Hi Cory,
>
> Actually it seems that the database wasn't shut down properly, and
> while it was doing the recovery, it refused connections. Expected and
> what needs to happen, indeed.
>
> That's why if you are a client about to connect, you'll recieve the "
> FATAL ...database is starting up " message.
>
> Ideally, you (the dba) should let the server end this task (the
> recovery). AFAIK, the recovery started as a consecuence of dirty data
> somewere, probably a big transaction not comitted when the server went
> off.
>
> Are you aware of a shutdown prior to this messages?
>
> Nothing to do with your current issue, but you will receive comments
> related with your pgsql version if you are not using a  pgsql 8 :)

As mentioned, this is normal operation.

Note that the command:

pg_ctl -m immediate stop

will cause a recovery.  It's pretty much a functional equivalent of
senting kill -9 to all backends and the postmaster at once.  Brutal but
effective.

Indeed, from the pg_ctl man pages:

``Immediate'' mode will abort all server processes without a clean
shutdown. This will lead to a recovery run on restart.

Regards,
Guido Barosio

On 3/14/06, Scott Marlowe <smarlowe@g2switchworks.com> wrote:
> On Tue, 2006-03-14 at 14:05, Guido Barosio wrote:
> > Hi Cory,
> >
> > Actually it seems that the database wasn't shut down properly, and
> > while it was doing the recovery, it refused connections. Expected and
> > what needs to happen, indeed.
> >
> > That's why if you are a client about to connect, you'll recieve the "
> > FATAL ...database is starting up " message.
> >
> > Ideally, you (the dba) should let the server end this task (the
> > recovery). AFAIK, the recovery started as a consecuence of dirty data
> > somewere, probably a big transaction not comitted when the server went
> > off.
> >
> > Are you aware of a shutdown prior to this messages?
> >
> > Nothing to do with your current issue, but you will receive comments
> > related with your pgsql version if you are not using a  pgsql 8 :)
>
> As mentioned, this is normal operation.
>
> Note that the command:
>
> pg_ctl -m immediate stop
>
> will cause a recovery.  It's pretty much a functional equivalent of
> senting kill -9 to all backends and the postmaster at once.  Brutal but
> effective.
>


--
/"\   ASCII Ribbon Campaign  .
\ / - NO HTML/RTF in e-mail  .
 X  - NO Word docs in e-mail .
/ \ -----------------------------------------------------------------

Ok, thanks a lot guys. 

It seems that I should have brought the database down to "local access" mode via pg_hba.conf and allow all transactions to fully recover before starting up replication or allowing other applications to access the db.

Have a good night all!

--
Cory Vokey