Kieren Scott <kierenscott@hotmail.com> wrote:
> What would be the best course of action for resolving a situation
> whereby your postgres instance had crashed due to the wal disk and
> archive wal disk becoming 100% full? Say your backups have been
> failing and your 'monitoring' had not reported it correctly.
>
> You can't start the instance because it needs to write to the WAL
> disk (which is full), but if you manually move WAL files off the
> WAL disk, the archiver will fail because it can't find WAL files
> it needs to archive. The instance may also still be in backup
> mode, because the backups had not completed due to the disk full
> situation.
>
> Being new to postgres, im trying to understand what actions need
> to be taken to get the instance back up and running without
> compromising recoverability...?
You will get more detailed advice if you avoid hypotheticals and say
exactly what's going on and what your priorities are. For starters,
are you OK with a situation which gets your primary database running
again and lets you start over with a new base backup, or is it
critical that you continue your backup stream without having to take
a new base backup? My advice would depend on that answer to that.
Also, it would be helpful to have an idea what your various mount
points are, how big they are, and what's on them. (If there's
something else *also* on the same mount point as the WAL files, that
might make a difference. What do you mean, exactly, when you say
your wal disk and archive wal disk are 100% full? (Are those
separate mount points? Did the archive fail to restore, thereby
building up to where the archive later began to fail, or is it a
shared drive?)
-Kevin