Re: Another PANIC corrupt index/crash ...any thoughts?

On Mon, Feb 1, 2010 at 7:45 AM, Jeff Amiel <becauseimjeff@yahoo.com> wrote:
> About a month ago I posted about a database crash possibly caused by corrupt index..
> Coincidentally (or not) started getting disk errors about a minute AFTER the above error (db storage is on a fibre
attachedSAN) 

Not likely a coincidence.

> /var/log/archive/log-2010-01-29.log:Jan 29 15:18:50 db-1 scsi_vhci: [ID 734749 kern.warning] WARNING: vhci_scsi_reset
0x1
> /var/log/archive/log-2010-01-29.log:Jan 29 15:18:50 db-1 scsi: [ID 243001 kern.warning] WARNING:
/pci@0,0/pci10de,5d@d/pci1077,142@0/fp@0,0(fcp1): 
> /var/log/archive/log-2010-01-29.log:Jan 29 15:18:52 db-1 scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk@g000b08001c001958(sd9): 
> /var/log/archive/log-2010-01-29.log:Jan 29 15:18:52 db-1 scsi: [ID 107833 kern.notice]  Requested Block: 206265378  
             Error Block: 206265378 
> /var/log/archive/log-2010-01-29.log:Jan 29 15:18:52 db-1 scsi: [ID 107833 kern.notice]  Vendor: Pillar              
             Serial Number: 
> /var/log/archive/log-2010-01-29.log:Jan 29 15:18:52 db-1 scsi: [ID 107833 kern.notice]  Sense Key: Unit Attention
> /var/log/archive/log-2010-01-29.log:Jan 29 15:18:52 db-1 scsi: [ID 107833 kern.notice]  ASC: 0x29 (power on, reset,
orbus reset occurred), ASCQ: 0x0, FRU: 0x0 
>
> Any thoughts on how I should proceed?

Figure out what's broken in your hardware?  It looks like a driver issue to me.

> We are planning an upgrade to 8.4 in the short-term, but I can see no evidence of fixes since the 8.2 version that
wouldrelate to index corruption. 

This is not a postgresql issue, it is a bad hardware / driver issue.
PostgreSQL cannot cause a SCSI reset etc on its own, it requires
something be broken in the OS / hardware for that to happen.

> I have no real evidence of bad disks...iostat -E reports:

No, but this iostat output is evidence of a bad SAN driver / SAN or
something around there.

>
> # iostat -E
> sd2       Soft Errors: 1 Hard Errors: 4 Transport Errors: 0
> Vendor: Pillar   Product: Axiom 300        Revision: 0000 Serial No:
> Size: 2.20GB <2200567296 bytes>
> Media Error: 0 Device Not Ready: 0 No Device: 4 Recoverable: 0
> Illegal Request: 1 Predictive Failure Analysis: 0
> sd3       Soft Errors: 1 Hard Errors: 32 Transport Errors: 0
> Vendor: Pillar   Product: Axiom 300        Revision: 0000 Serial No:
> Size: 53.95GB <53948448256 bytes>
> Media Error: 0 Device Not Ready: 0 No Device: 32 Recoverable: 0
> Illegal Request: 1 Predictive Failure Analysis: 0
> sd7       Soft Errors: 1 Hard Errors: 40 Transport Errors: 8
> Vendor: Pillar   Product: Axiom 300        Revision: 0000 Serial No:
> Size: 53.95GB <53948448256 bytes>
> Media Error: 0 Device Not Ready: 1 No Device: 33 Recoverable: 0
> Illegal Request: 1 Predictive Failure Analysis: 0
> sd8       Soft Errors: 1 Hard Errors: 34 Transport Errors: 0
> Vendor: Pillar   Product: Axiom 300        Revision: 0000 Serial No:
> Size: 107.62GB <107622432256 bytes>
> Media Error: 0 Device Not Ready: 0 No Device: 34 Recoverable: 0
> Illegal Request: 1 Predictive Failure Analysis: 0
> sd9       Soft Errors: 1 Hard Errors: 32 Transport Errors: 2
> Vendor: Pillar   Product: Axiom 300        Revision: 0000 Serial No:
> Size: 215.80GB <215796153856 bytes>
> Media Error: 0 Device Not Ready: 1 No Device: 29 Recoverable: 0
> Illegal Request: 1 Predictive Failure Analysis: 0