Обсуждение: Error: pid file is invalid, please manually kill the stale server process.

Hi all,

I am using postgres on Ubuntu 12.04 64 bit server. The version is 9.1.10
and I used the stock postgres from Ubuntu repos.

I have a weird problem. Sometimes when the machine is restarted, there
is no $PGDATA/postmaster.pid file. So when I try to restart postgres
(service postgresql restart) I get the following error:

Error: pid file is invalid, please manually kill the stale server process.

An even scarier side-effect of this is that the data, when this happens,
are older than the most recent data in the database.

It may be important to know that postgres runs under monit non sysv init
or upstart.

Also postgres is installed on a customer appliance that may experience
frequent reboots and even abrupt poweroffs! Is postgres in general ideal
for this situation? Is there an optimal configuration for this scenario?

Regards

theo

Theo,

Don't know much about the tools you may have installed on Ubuntu (don't use Ubuntu), but sounds like you have something
'snapshot-ing'your dataset... And that, on restart-after-inelegant-shutdown, the PostgreSQL service is trying to attach
tothe 'wrong' data. 

Generally, yes; PostgreSQL is extremely reliable. We've found it to recover quite nicely, even from bad (human,
sometimestesting-intentional!) behavior. Also doesn't sound like a version-specific problem. 

Lou Picciano

----- Original Message -----
From: "Theodotos Andreou" <theo@theo-andreou.org>
To: pgsql-admin@postgresql.org
Sent: Wednesday, February 12, 2014 12:59:22 AM
Subject: [ADMIN] Error: pid file is invalid, please manually kill the stale server process.

Hi all,

I am using postgres on Ubuntu 12.04 64 bit server. The version is 9.1.10
and I used the stock postgres from Ubuntu repos.

I have a weird problem. Sometimes when the machine is restarted, there
is no $PGDATA/postmaster.pid file. So when I try to restart postgres
(service postgresql restart) I get the following error:

Error: pid file is invalid, please manually kill the stale server process.

An even scarier side-effect of this is that the data, when this happens,
are older than the most recent data in the database.

It may be important to know that postgres runs under monit non sysv init
or upstart.

Also postgres is installed on a customer appliance that may experience
frequent reboots and even abrupt poweroffs! Is postgres in general ideal
for this situation? Is there an optimal configuration for this scenario?

Regards

theo


--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

On Wed, Feb 12, 2014 at 6:59 AM, Theodotos Andreou
<theo@theo-andreou.org> wrote:
> I have a weird problem. Sometimes when the machine is restarted, there is no
> $PGDATA/postmaster.pid file. So when I try to restart postgres (service
> postgresql restart) I get the following error:
>

It is not clear what the real problem is. After a machine reboot it is
normal that there is no pid file, meaning a clean shutdown happened
(from the PostgreSQL perspective). After a full start, if the
PostgreSQL process is configured for automatic start, a pid file
should exist. I suspect the problem is related to some ubuntu booting
script that will place the pid file somewhere else from where the
pg_ctl expects to find it.

> An even scarier side-effect of this is that the data, when this happens, are
> older than the most recent data in the database.
>

This does not make sense at all. How do you know that data is older?
And most notably: is data stable? I mean, could it be some kind of
snapshotting of the database filesystem (as other suggested) or sounds
like a data corruption?


> Also postgres is installed on a customer appliance that may experience
> frequent reboots and even abrupt poweroffs! Is postgres in general ideal for
> this situation? Is there an optimal configuration for this scenario?

PostgreSQL is reliable, but even a reliable system can crash when
running on a totally unreliable piece of hardware. I suggest investing
in some good platform to avoid stressing the database with resumes.
However, in such scenario, I will choose a sync of all commits  and
short checkpoints, but this will lead to bad performances.

Luca

Guy thanks for your replies.

You are right about the strange

I found the problem and is related to the peculiar setup we have on this
appliance. PGDATA exists on a mounted partition but we somehow managed
to create an older version of PGDATA under the mount point.

So sometimes postgres would load the data from the unmounted PGDATA and
sometimes from the mounted partition.

It' s a pretty unlikely scenario and I burned lots of braincells trying
to figure it out.

So not really a postgres problem but rather our peculiar setup.

On 02/13/2014 10:39 AM, Luca Ferrari wrote:
> On Wed, Feb 12, 2014 at 6:59 AM, Theodotos Andreou
> <theo@theo-andreou.org> wrote:
>> I have a weird problem. Sometimes when the machine is restarted, there is no
>> $PGDATA/postmaster.pid file. So when I try to restart postgres (service
>> postgresql restart) I get the following error:
>>
> It is not clear what the real problem is. After a machine reboot it is
> normal that there is no pid file, meaning a clean shutdown happened
> (from the PostgreSQL perspective). After a full start, if the
> PostgreSQL process is configured for automatic start, a pid file
> should exist. I suspect the problem is related to some ubuntu booting
> script that will place the pid file somewhere else from where the
> pg_ctl expects to find it.
>
>> An even scarier side-effect of this is that the data, when this happens, are
>> older than the most recent data in the database.
>>
> This does not make sense at all. How do you know that data is older?
> And most notably: is data stable? I mean, could it be some kind of
> snapshotting of the database filesystem (as other suggested) or sounds
> like a data corruption?
>
>
>> Also postgres is installed on a customer appliance that may experience
>> frequent reboots and even abrupt poweroffs! Is postgres in general ideal for
>> this situation? Is there an optimal configuration for this scenario?
> PostgreSQL is reliable, but even a reliable system can crash when
> running on a totally unreliable piece of hardware. I suggest investing
> in some good platform to avoid stressing the database with resumes.
> However, in such scenario, I will choose a sync of all commits  and
> short checkpoints, but this will lead to bad performances.
>
> Luca
>
>

On Thu, Feb 13, 2014 at 11:09 AM, Theodotos Andreou
<theo@theo-andreou.org> wrote:
> So sometimes postgres would load the data from the unmounted PGDATA and
> sometimes from the mounted partition.

Stackable mount points....I had to fight against them for a while!
It happened to me to have the same problem a lot of times, even if not
related to database.

Good luck.

Luca