Re: Issue with rsync based incremental backup : postgres: startup process waiting for 0000000100000001000000D2

It worked! 

I had even setup streaming replication after doing incremental replication, without needing to stop postgres on the primary server.

Here is the script i came up with :

#!/bin/bash
if [ $# -ne 1 ]; then
echo "you must specify the hostname of the backup server";
exit 0;
fi;
BACKUP_SERVER=$1
PGDATA=/var/lib/pgsql/data
PGXLOG=$PGDATA/pg_xlog
PGEXEC="sudo su -l postgres -s /bin/bash -c"
RSYNC="rsync"
OPTIONS="--archive --checksum --compress --progress"
EXCLUDES="--exclude postmaster.pid --exclude postgresql.conf --exclude pg_hba.conf --exclude server.crt --exclude server.key"
ROLLOVER=32
SSH="ssh -q -o StrictHostKeyChecking=no -o BatchMode=yes $BACKUP_SERVER"
REPLICATION_CHECK="$SSH ps aux | grep postgres | grep wal | grep receiver"
#On BACKUP_SERVER
if [ -n "$(service postgresql status | grep "pid[:blank:]*[0-9]*")" ]; then 
$SSH "service postgresql stop"
fi;
#On PRIMARY
echo "Running VACUUM"
$PGEXEC "psql -c \"VACUUM FULL;\""
echo "VACUUM completed"

for f in $(ls -tr $PGXLOG | head -n ${ROLLOVER}); do 
$RSYNC $OPTIONS $PGXLOG/$f $BACKUP_SERVER:$PGXLOG/
done;
$PGEXEC "psql -c \"SELECT pg_start_backup('incremental_backup',true);\""
$RSYNC $OPTIONS $EXCLUDES --exclude pg_xlog $PGDATA $BACKUP_SERVER:$PGDATA
$PGEXEC "psql -c \"SELECT pg_stop_backup();\""
$RSYNC $OPTIONS $PGXLOG $BACKUP_SERVER:$PGXLOG
$RSYNC $OPTIONS $PGXLOG $BACKUP_SERVER:$PGXLOG
#On BACKUP_SERVER
$SSH "service postgresql start"
if [ -z "$(service postgresql status | grep "pid[:blank:]*[0-9]*")" ]; then 
echo "Failed to start database on backup server"
        echo "Look into the postgres logs for more details"
        echo "exiting..."
        exit 1;
fi;
#need to improve this delay-check to wait until the backup server has finished recovery and started into streaming mode
sleep 30
if [ -n "$(${REPLICATION_CHECK})" ] ; then
echo "SUCCESS in synching BACKUP_SERVER with the latest data from Primary"; 
#On BACKUP_SERVER
$SSH "service postgresql stop"
echo "Stopped the backup server in good state; it will get updated in the next scheduled incremental backup"
else 
echo "FAILED to sync backup server with Primary";
        echo "Leaving the backup server running in the failed state for further debugging"
        exit 1; 
fi;
exit 0;