Forgot to mention we have streaming replication partner as well that around the same time stop recovering because a WAL segment could not be found on the primary.
On Sat, Jul 16, 2011 at 4:05 AM, Ken Caruso
<ken@ipl31.net> wrote:
Hello,
Had a DB that had bloated due to lots of updates and insuffcient auto vacuum settings. Attempted the following in the following order:
VACUUM VERBOSE full tablename;
CLUSTER tablename;
Did the full first to ensure that the cluster had enough space to run.
The db had already been previously clustered to an index. At some point during the VACUUM I started seeing the following errors in the logs:
CONTEXT: writing block 12125253 of relation base/2651908/652397108
WARNING: could not write block 12125253 of base/2651908/652397108
DETAIL: Multiple failures --- write error might be permanent.
Also seeing the same error for block 12125253, 2978.
And the cluster failed as well. I have attempted a check point using pg_start_backup() and that failed as well.
The system is still running and I have yet to shut it down. I am hoping to find out what possibilities I have if any to recover from this. Also how do I avoid this in the future?
-Ken