Re: full data disk -- any chance of recovery

You wrote:
>
> Greg,
>
> Does pg_ctl stop -m immediate stop the postmaster for you?

I tried
su - postgres -c '/apps/pgsql-7.4/bin/pg_ctl stop -D /data/postgres/gex_runtime -m immediate'

on one of the two hozed servers and that's (I think) what got this:

2006-01-01 23:20:19 WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because
anotherserver process exited abnormally and possibly corrupted shared memory. 
HINT:  In a moment you should be able to reconnect to the database and repeat your command.

And that's been like that for a while

While the other server (unstopped) shows only:

2006-01-02 00:30:01 LOG:  could not close temporary statistics file
"/data/postgres/gex_runtime/global/pgstat.tmp.1453":No space left on device 
2006-01-02 00:33:54 ERROR:  could not access status of transaction 0
DETAIL:  could not write to file "/data/postgres/gex_runtime/pg_clog/0AFA" at offset 196608: No space left on device

G


----
Jeff Frost, Owner       <jeff@frostconsultingllc.com>
Frost Consulting, LLC   http://www.frostconsultingllc.com/
Phone: 650-780-7908     FAX: 650-649-1954


-----Original Message-----
From: pgsql-admin-owner@postgresql.org
[mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Gregory S. Williamson
Sent: Sunday, January 01, 2006 11:58 PM
To: Jeff Frost; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] full data disk -- any chance of recovery

Jeff --

Thanks for the suggestion -- I think this fills the bill except that the
postmaster won't quit because it has no space (at least that is how I
interpet it). These are all linux boxes with the same architecture (2 CPUs,
2 gigs of RAM, disks not adequate for a database: QED).

I had an urgent priority in November to upgrade these beasts, but the best
laid plans o' mice and men, etc. etc.

These servers are mostly read-only for spatial data so falling back to the
last known state (e.g. before the current transaction) would work perfectly.

But I'm still making a copy o' one of the two hot spares (one of which is
now in play), juts in case.

Have a good {day|afternoon|evening|night) !

Greg

-----Original Message-----
From:    jeff@glacier.frostconsultingllc.com on behalf of Jeff Frost
Sent:    Sun 1/1/2006 11:49 PM
To:    Gregory S. Williamson; pgsql-admin@postgresql.org
Cc:
Subject:    RE: [ADMIN] full data disk -- any chance of recovery
Greg, I'm not sure what you're looking for in the way of suggestions.  Do
you just want to be able to start this postgres server up and remove some
data?  Easiest way I see to accomplish that given the information you
provided is to move pg_xlog to the sda disk and symlink it to the data dir.


In general terms, it would go like this:

Stop postmaster
cd /data/gex_runtime
mv pg_xlog /
ln -s /pg_xlog
Start postmaster

The commands may vary depending on OS.

That would also give you better performance if sda and sdb are actually
separate physical disks.  However, that's only going to give you about 500MB
of free space, so I see bigger disks in your future.  A vacuum full might
recover a bit of space as well if you've got any bloat.

The question I have is this: Is your database read-only?  Otherwise,
bringing these machines back up probably isn't too useful as they are now
out of sync with the new primary (your old hot spare).

Good luck!

-----Original Message-----
From: pgsql-admin-owner@postgresql.org
[mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Gregory S. Williamson
Sent: Sunday, January 01, 2006 11:28 PM
To: pgsql-admin@postgresql.org
Subject: [ADMIN] full data disk -- any chance of recovery

Availables space is:
$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             32850580   3137552  28044280  11% /
/dev/sdb1             35001508  33223500        16 100% /data

Any suggestions ? Falling back to the last known state is fine, but just in
case I am making a backup of the remaining database to build a replacement.

And yes, I did forsee this and did warn management repeatedly and yet
somehow the advice falls on deaf ears. Go figure. I guess maybe because it
isn't management that a hole kicked in a 3 day weekend.









---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend



!DSPAM:43b8e0cf33531348188260!