Re: PANIC: heap_update_redo: no block

<snip>

> Hrm ok ill see about doing either the patch or setting wal_debug to
> true (or both).  However im currently on vacation till Saturday so ill
> do this first thing then and report the results back.  Thank you very
> much!

Ok here is the output with wal_debug set to 1 in the config and the
final supplied patch by Qingqing.

(@)<2006-03-25 20:54:17.509 MST>[26571]LOG:  database system was
interrupted while in recovery at 2006-03-25 20:51:58 MST
(@)<2006-03-25 20:54:17.509 MST>[26571]HINT:  This probably means that
some data is corrupted and you will have to use the last backup for
recovery.
(@)<2006-03-25 20:54:17.509 MST>[26571]LOG:  checkpoint record is at D/1919D5F0
(@)<2006-03-25 20:54:17.509 MST>[26571]LOG:  redo record is at
D/191722C8; undo record is at 0/0; shutdown FALSE
(@)<2006-03-25 20:54:17.509 MST>[26571]LOG:  next transaction ID:
81148900; next OID: 16566476
(@)<2006-03-25 20:54:17.509 MST>[26571]LOG:  next MultiXactId: 1; next
MultiXactOffset: 0
(@)<2006-03-25 20:54:17.509 MST>[26571]LOG:  database system was not
properly shut down; automatic recovery in progress
(@)<2006-03-25 20:54:17.524 MST>[26571]LOG:  redo starts at D/191722C8
(@)<2006-03-25 20:54:17.524 MST>[26571]LOG:  REDO @ D/191722C8; LSN
D/19172678: prev D/191722A0; xid 81148908: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.524 MST>[26571]LOG:  REDO @ D/19172678; LSN
D/191726B8: prev D/191722C8; xid 81148908: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.524 MST>[26571]LOG:  REDO @ D/191726B8; LSN
D/19172788: prev D/19172678; xid 81148908: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.524 MST>[26571]LOG:  REDO @ D/19172788; LSN
D/191727C8: prev D/191726B8; xid 81148908: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.524 MST>[26571]LOG:  REDO @ D/191727C8; LSN
D/19172A30: prev D/19172788; xid 81148908: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.524 MST>[26571]LOG:  REDO @ D/19172A30; LSN
D/19172A70: prev D/191727C8; xid 81148908: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.524 MST>[26571]LOG:  REDO @ D/19172A70; LSN
D/19172F68: prev D/19172A30; xid 81148908: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.524 MST>[26571]LOG:  REDO @ D/19172F68; LSN
D/19172FA8: prev D/19172A70; xid 81148908: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.524 MST>[26571]LOG:  REDO @ D/19172FA8; LSN
D/19172FD0: prev D/19172F68; xid 81148908: Transaction - commit:
2006-03-18 22:51:5
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19172FD0; LSN
D/19173088: prev D/19172FA8; xid 81148942: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173088; LSN
D/191730C8: prev D/19172FD0; xid 81148942: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/191730C8; LSN
D/191731E8: prev D/19173088; xid 81148942: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/191731E8; LSN
D/19173228: prev D/191730C8; xid 81148942: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173228; LSN
D/19173250: prev D/191731E8; xid 81148942: Transaction - commit:
2006-03-18 22:51:5
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173250; LSN
D/19173414: prev D/19173228; xid 81148944: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173414; LSN
D/19173454: prev D/19173250; xid 81148944: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173454; LSN
D/19173760: prev D/19173414; xid 81148944: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173760; LSN
D/191737A0: prev D/19173454; xid 81148944: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/191737A0; LSN
D/191737C8: prev D/19173760; xid 81148944: Transaction - commit:
2006-03-18 22:51:5
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/191737C8; LSN
D/1917380C: prev D/191737A0; xid 81148946: Heap - update: rel
1663/16386/16865; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/1917380C; LSN
D/19173834: prev D/191737C8; xid 81148946: Transaction - commit:
2006-03-18 22:51:5
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173834; LSN
D/191739B8: prev D/1917380C; xid 81148948: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/191739B8; LSN
D/191739F8: prev D/19173834; xid 81148948: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/191739F8; LSN
D/19173AE4: prev D/191739B8; xid 81148948: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173AE4; LSN
D/19173B24: prev D/191739F8; xid 81148948: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173B24; LSN
D/19173B4C: prev D/19173AE4; xid 81148948: Transaction - commit:
2006-03-18 22:51:5
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173B4C; LSN
D/19173C38: prev D/19173B24; xid 81148960: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173C38; LSN
D/19173C78: prev D/19173B4C; xid 81148960: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.525 MST>[26571]LOG:  REDO @ D/19173C78; LSN
D/19173EFC: prev D/19173C38; xid 81148960: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19173EFC; LSN
D/19173F3C: prev D/19173C78; xid 81148960: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19173F3C; LSN
D/19173F64: prev D/19173EFC; xid 81148960: Transaction - commit:
2006-03-18 22:51:5
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19173F64; LSN
D/19174064: prev D/19173F3C; xid 81148962: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19174064; LSN
D/191740A4: prev D/19173F64; xid 81148962: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/191740A4; LSN
D/19174294: prev D/19174064; xid 81148962: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19174294; LSN
D/191742D4: prev D/191740A4; xid 81148962: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/191742D4; LSN
D/191742FC: prev D/19174294; xid 81148962: Transaction - commit:
2006-03-18 22:51:5
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/191742FC; LSN
D/191743E8: prev D/191742D4; xid 81148964: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/191743E8; LSN
D/19174428: prev D/191742FC; xid 81148964: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19174428; LSN
D/191746FC: prev D/191743E8; xid 81148964: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/191746FC; LSN
D/1917473C: prev D/19174428; xid 81148964: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/1917473C; LSN
D/19174764: prev D/191746FC; xid 81148964: Transaction - commit:
2006-03-18 22:51:5
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19174764; LSN
D/19174824: prev D/1917473C; xid 81148966: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19174824; LSN
D/19174864: prev D/19174764; xid 81148966: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19174864; LSN
D/191749A4: prev D/19174824; xid 81148966: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/191749A4; LSN
D/191749E4: prev D/19174864; xid 81148966: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/191749E4; LSN
D/19174A0C: prev D/191749A4; xid 81148966: Transaction - commit:
2006-03-18 22:51:5
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19174A0C; LSN
D/19174D18: prev D/191749E4; xid 81148968: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19174D18; LSN
D/19174D58: prev D/19174A0C; xid 81148968: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.526 MST>[26571]LOG:  REDO @ D/19174D58; LSN
D/19174E60: prev D/19174D18; xid 81148968: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19174E60; LSN
D/19174EA0: prev D/19174D58; xid 81148968: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19174EA0; LSN
D/19174EC8: prev D/19174E60; xid 81148968: Transaction - commit:
2006-03-18 22:52:0
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19174EC8; LSN
D/1917508C: prev D/19174EA0; xid 81148970: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/1917508C; LSN
D/191750CC: prev D/19174EC8; xid 81148970: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/191750CC; LSN
D/19175194: prev D/1917508C; xid 81148970: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19175194; LSN
D/191751D4: prev D/191750CC; xid 81148970: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/191751D4; LSN
D/191752C8: prev D/19175194; xid 81148970: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/191752C8; LSN
D/19175308: prev D/191751D4; xid 81148970: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19175308; LSN
D/191753D0: prev D/191752C8; xid 81148970: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/191753D0; LSN
D/19175410: prev D/19175308; xid 81148970: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19175410; LSN
D/19175438: prev D/191753D0; xid 81148970: Transaction - commit:
2006-03-18 22:52:0
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19175438; LSN
D/19175524: prev D/19175410; xid 81148972: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19175524; LSN
D/19175564: prev D/19175438; xid 81148972: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19175564; LSN
D/19175844: prev D/19175524; xid 81148972: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19175844; LSN
D/19175884: prev D/19175564; xid 81148972: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19175884; LSN
D/1917596C: prev D/19175844; xid 81148972: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/1917596C; LSN
D/191759AC: prev D/19175884; xid 81148972: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/191759AC; LSN
D/191759D4: prev D/1917596C; xid 81148972: Transaction - commit:
2006-03-18 22:52:0
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/191759D4; LSN
D/19175B98: prev D/191759AC; xid 81148974: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19175B98; LSN
D/19175BD8: prev D/191759D4; xid 81148974: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.527 MST>[26571]LOG:  REDO @ D/19175BD8; LSN
D/19175EE4: prev D/19175B98; xid 81148974: Heap - update: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/19175EE4; LSN
D/19175F24: prev D/19175BD8; xid 81148974: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/19175F24; LSN
D/19175F4C: prev D/19175EE4; xid 81148974: Transaction - commit:
2006-03-18 22:52:0
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/19175F4C; LSN
D/19176104: prev D/19175F24; xid 81148976: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/19176104; LSN
D/19176144: prev D/19175F4C; xid 81148976: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/19176144; LSN
D/19176230: prev D/19176104; xid 81148976: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/19176230; LSN
D/19176270: prev D/19176144; xid 81148976: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/19176270; LSN
D/19176298: prev D/19176230; xid 81148976: Transaction - commit:
2006-03-18 22:52:0
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/19176298; LSN
D/19176384: prev D/19176270; xid 81148978: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/19176384; LSN
D/191763C4: prev D/19176298; xid 81148978: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/191763C4; LSN
D/191765A8: prev D/19176384; xid 81148978: Heap - insert: rel
1663/16386/2619; tid
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/191765A8; LSN
D/191765E8: prev D/191763C4; xid 81148978: Btree - insert: rel
1663/16386/2696; tid
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/191765E8; LSN
D/19176610: prev D/191765A8; xid 81148978: Transaction - commit:
2006-03-18 22:52:0
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/19176610; LSN
D/19176644: prev D/191765E8; xid 81148979: Heap - clean: rel
1663/16386/16559898; b
(@)<2006-03-25 20:54:17.528 MST>[26571]LOG:  REDO @ D/19176644; LSN
D/191766A4: prev D/19176610; xid 81148979: Heap - move: rel
1663/16386/16559898; ti
(@)<2006-03-25 20:54:17.528 MST>[26571]PANIC:  heap_update_redo: no
block: target blcknum: 1, relation(1663/16386/16559898) length: 1
(@)<2006-03-25 20:54:17.529 MST>[26570]LOG:  startup process (PID
26571) was terminated by signal 6
(@)<2006-03-25 20:54:17.530 MST>[26570]LOG:  aborting startup due to
startup process failure

Ok so whats next?