Re: checkpoint_timeout and archive_timeout

Jeff Janes <jeff.janes@gmail.com> wrote:
> Kevin Grittner <Kevin.Grittner@wicourts.gov> wrote:

>> We pipe the file into pg_clearxlogtail | gzip and pipe it out to
>> the archive directory (with a ".gz" suffix), rather than using cp
>> and processing it later.  Well, actually, we pipe it to a
>> directory on the same mount point as the archive directory and mv
>> it into place, as part of our scheme to avoid problems with
>> partial files.
>
> Do you have an example of that which you could share?
>
> I've run into two problems I'm trying to overcome.  One is that
> pg_clearxlogtail fails on file formats it doesn't recognize but is
> asked to archive anyway, such as
> '000000010000000200000065.00000020.backup', for example.  Perhaps
> it could just issue a warning and then pass the unrecognized file
> through unchanged, instead of bailing out with a fatal error.
>
> The other is that a pipeline in bash reports success even if an
> interior member of it failed.  I think I know how to fix that
> under an actual shell script (as opposed to a pipeline stuffed
> into "archive_command"), but would like to see how other people
> have dealt with it.

I make no claims that this couldn't be improved upon, but it has
been humming along without any problems without modification for
about five years now on 80-some machines.  Our archive_command is
just a call to a bash script that looks like this:


walPathname=$1
walFilename=$2

tempfile=/var/pgsql/data/wal-files-temp/$walFilename.gz
targetfile=/var/pgsql/data/wal-files/$walFilename.gz
errfile=/home/ccsa/archive.err
emailfile=/home/ccsa/$walFilename.rpt

cat <<END > runtime
walPathname=$walPathname
walFilename=$walFilename
tempfile=$tempfile
targetfile=$targetfile
errfile=$errfile
emailfile=$emailfile
END

if [ -f $targetfile ] ; then
  echo "Cannot copy $walPathName to $targetfile \
because it already exists" 2> $errfile
  archivereturnval=3
else
  if echo $walFilename | grep -Eq ^[0-9A-F]{24}$ - ; then
    pg_clearxlogtail < $walPathname 2> $errfile \
      | gzip > $tempfile 2>> $errfile \
      && mv $tempfile $targetfile 2>> $errfile
    archivereturnval=$?
  else
    gzip < $walPathname > $tempfile 2> $errfile \
      && mv $tempfile $targetfile 2>> $errfile
    archivereturnval=$?
  fi
fi

if [ "$archivereturnval" -eq "0" ] ; then
  # Successful archive.
  # Delete any email file from a possible prior failure.
  rm -f $emailfile $errfile
  exit 0
fi

# Failed to archive.
if [ ! -f $emailfile ] ; then
  # No existing email files -- make one and send it.
  echo "date: $(date +%F_%H%M)" > $emailfile
  echo "to: dba.group@wicourts.gov" >> $emailfile
  echo "subject: Problem with archiving $walFilename ">> $emailfile
  echo " "  >> $emailfile
  echo "$archivereturnval return code from archive" >> $emailfile
  echo " "  >> $emailfile
  cat $errfile >> $emailfile
  echo " "  >> $emailfile
  cat runtime >> $emailfile
  /usr/sbin/sendmail -t < $emailfile
fi
cat $errfile > /dev/stderr
exit $archivereturnval