Hey guys,
I have a problem i’m stuck on. Our Postgres server ran out of disk space last night and it fell over. We’ve resolved
thedisk space issue and there is plenty now although Postgres will not start.
When I view the main log it reads:
2015-02-23 17:49:35 UTC LOG: redo starts at 75/202674D8
2015-02-23 17:49:35 UTC LOG: incomplete startup packet
2015-02-23 17:49:36 UTC FATAL: the database system is starting up
2015-02-23 17:49:36 UTC FATAL: the database system is starting up
2015-02-23 17:49:41 UTC FATAL: the database system is starting up
2015-02-23 17:49:41 UTC LOG: incomplete startup packet
When I look at ps , I can see that postgres is trying to perform a recovery, although it never finishes.
- postgres: startup process recovering 00000001000000750000002D
I ran an strace on this process to try and see what it’s doing and I see these lines repeating over and over again:
read(7, 0x7fff6c82fadf, 1) = -1 EAGAIN (Resource temporarily unavailable)
open("pg_clog/01ED", O_RDWR|O_CREAT, 0600) = 27
lseek(27, 212992, SEEK_SET) = 212992
write(27, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192
fsync(27) = 0
close(27)
When I look under pg_clog I can see that the 01ED file it’s opening here is the most recent one and has a different
filesize to the others in the dir.
I’m assuming this is the file it was writing too when the disk space ran out and now it’s corrupt or something along
thoselines.
Can anyone please help me figure out how to fix this? I was told that deleting any of these pg_clog files would be
bad,how ever I had no idea how to repair it.
p.s I also asked this question
here: http://dba.stackexchange.com/questions/93594/postgres-wont-start-incomplete-startup-packet
Cheers
Cheyne