Re: Disaster!

pg_clog information:

# cd pg_clog
# ls -al
total 3602
drwx------  2 pgsql  pgsql     512 Jan 23 03:49 .
drwx------  6 pgsql  pgsql     512 Jan 23 12:30 ..
-rw-------  1 pgsql  pgsql  262144 Jan 18 19:43 0000
-rw-------  1 pgsql  pgsql  262144 Jan 18 19:43 0001
-rw-------  1 pgsql  pgsql  262144 Jan 18 19:43 0002
-rw-------  1 pgsql  pgsql  262144 Jan 18 19:43 0003
-rw-------  1 pgsql  pgsql  262144 Jan 19 08:35 0004
-rw-------  1 pgsql  pgsql  262144 Jan 21 10:35 0005
-rw-------  1 pgsql  pgsql  262144 Jan 20 10:07 0006
-rw-------  1 pgsql  pgsql  262144 Jan 21 19:30 0007
-rw-------  1 pgsql  pgsql  262144 Jan 21 17:24 0008
-rw-------  1 pgsql  pgsql  262144 Jan 22 06:50 0009
-rw-------  1 pgsql  pgsql  262144 Jan 22 13:01 000A
-rw-------  1 pgsql  pgsql  262144 Jan 23 06:45 000B
-rw-------  1 pgsql  pgsql  262144 Jan 23 09:37 000C
-rw-------  1 pgsql  pgsql  163840 Jan 23 12:30 000D

I don't have debug symbols, so backtrace isn't that helpful:

(gdb) bt
#0  0x2840a848 in kill () from /usr/lib/libc.so.4
#1  0x2844ee90 in abort () from /usr/lib/libc.so.4
#2  0x81b33ba in ?? ()
#3  0x8092a0c in ?? ()
#4  0x8092450 in ?? ()
#5  0x808a9d7 in ?? ()
#6  0x808adf4 in ?? ()
#7  0x808ae8c in ?? ()
#8  0x808bffa in ?? ()
#9  0x809082f in ?? ()
#10 0x8095204 in ?? ()
#11 0x8135c6b in ?? ()
#12 0x813309c in ?? ()
#13 0x8108dc3 in ?? ()
#14 0x806d49b in ?? ()

I have a backup of the data dir for future analysis, but this is kind of 
an urgent fix right now!  (Especially since it's 4am my time :( )

Thanks,

Chris

Christopher Kings-Lynne wrote:

> We ran out of disk space on our main server, and now I've freed up 
> space, we cannot start postgres!
> 
> Jan 23 12:18:50 canaveral postgres[563]: [2-1] LOG:  checkpoint record 
> is at 2/96500B94
> Jan 23 12:18:50 canaveral postgres[563]: [3-1] LOG:  redo record is at 
> 2/964BD23C; undo record is at 0/0; shutdown FALSE
> Jan 23 12:18:50 canaveral postgres[563]: [4-1] LOG:  next transaction 
> ID: 14216463; next OID: 4732327
> Jan 23 12:18:50 canaveral postgres[563]: [5-1] LOG:  database system was 
> not properly shut down; automatic recovery in progress
> Jan 23 12:18:50 canaveral postgres[563]: [6-1] LOG:  redo starts at 
> 2/964BD23C
> Jan 23 12:18:51 canaveral postgres[563]: [7-1] PANIC:  could not access 
> status of transaction 14286850
> Jan 23 12:18:51 canaveral postgres[563]: [7-2] DETAIL:  could not read 
> from file "/usr/local/pgsql/data/pg_clog/000D" at offset 163840: 
> Undefined error: 0
> Jan 23 12:18:51 canaveral postgres[567]: [1-1] FATAL:  the database 
> system is starting up
> Jan 23 12:18:52 canaveral postgres[558]: [1-1] LOG:  startup process 
> (PID 563) was terminated by signal 6
> Jan 23 12:18:52 canaveral postgres[558]: [2-1] LOG:  aborting startup 
> due to startup process failure
> 
> What can I do?
> 
> Chris
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if your
>      joining column's datatypes do not match