Обсуждение: Streaming replication: rsync to switchover

Hi,

I've setup streaming replication + file-based log-shipping, with a
hot-standby.

Everything's good, except I don't know which is the correct way to sync
the once-master db with the newly promoted master after a switchover,

1) I shutdown the master and touch the trigger_file on the slave. Now
the slave becomes the master.
2) On the old-master I create recovery.conf + archive directory for
file-based log-shipping.
3) I hope I can setup the old-master with a simple rsync, because the
operation could be really fast, being M and S almost identical, but I'm
not sure about the correct way to rsync them:

a) should I rsync --delete?
b) should I delete the pg_xlog of the old master, before rsyncing? (I'm
wondering what could happen if some of them never made its way to the
slave, that now has become new-master)
c) am I missing something? For example, I've read somewhere that one
should copy pg_control and pg_stat from the old slave to the new slave
but it doesn't make sense to me.


thanks for any hint,


--
rudi pettazzi

Hi, I do rsync -a --delete new_primary_server::postgresql-data/ /var/lib/postgresql/9.0/data/

Take care to use -a, I had -r and wondered, why the rsync took so long, despite the data was nearly the same.

assuming you have configured rsyncd on the server as:

[postgresql-data]
    uid = postgres
    path = /var/lib/postgresql/9.0/data
    comment = PostgreSQL 9.0 data dir
    exclude = postmaster.log pg_xlog/* postmaster.pid postgresql.conf

The exclude might be tunable a litte (the confs eg. are not stored in the datadir anymore on my distro)
Always rsync over the existing data dir (of course stop server before)
The former master can also be stopped later. When the trigger file is created, the connection to it is closed anyhow.
Don't forget to pg_start_backup pg_stop_backup on the new server before/after you do rsync.

Gerhard

On 04/19/2011 10:09 AM, Gerhard Hintermayer wrote:
> assuming you have configured rsyncd on the server as:
>
> [postgresql-data]
>      uid = postgres
>      path = /var/lib/postgresql/9.0/data
>      comment = PostgreSQL 9.0 data dir
>      exclude = postmaster.log pg_xlog/* postmaster.pid postgresql.conf

So you keep the old pg_xlog ... But, what about wal that the former
master hasn't yet sent to the slave? I wonder if the former-master could
apply them, becoming out-of-sync with the former-slave (that never
recevied those wal).

thanks

--
rudi

I also streaming replication _and_ ship WALs them to the slaves, something like: restore_command='rsync -pog master::postgresql-wals/%f %p', so normally the slaves are up to date (via streaming replication) and the WALs are also on the slaves to be able to recover (maybe to some PIT) from them in case something goes wrong.
The recommended setup should be handled more in detail in the docs, since this is a topic where you can make a lot of mistakes :-(

Gerhard

On Tue, Apr 19, 2011 at 11:16:08AM +0200, rudi wrote:
> On 04/19/2011 10:09 AM, Gerhard Hintermayer wrote:
> So you keep the old pg_xlog

the admin book says to exclude old pg_xlog

On 04/19/2011 02:11 PM, Ray Stell wrote:
> On Tue, Apr 19, 2011 at 11:16:08AM +0200, rudi wrote:
>> On 04/19/2011 10:09 AM, Gerhard Hintermayer wrote:
>> So you keep the old pg_xlog
>
> the admin book says to exclude old pg_xlog
>

Which book is this?

So I guess something like this (from new master to former master) should
be safe:

$ psql -c "SELECT pg_start_backup('switchover base backup')"
$ rsync -av /var/lib/postgresql/9.0/main/ --delete --exclude server.crt
--exclude server.key --exclude recovery.* --exclude postmaster.pid
--exclude archive $FORMER_MASTER_ADDRESS:/var/lib/postgresql/9.0/main/
$ psql -c "SELECT pg_stop_backup()"

What about the archiving directory of the former slave (now master)?
It should be cleaned guess, in case it will be slave again. If I look at
that directory, after a failover it is not empty.
This looks strange to me because pg_cleanup_archive should have already
cleaned it.

BTW, is there any documentation about the logic behind pg_xlog
filenames? It is really obscure and this makes really hard understanding
what's going on. I'm sure that most recent mtime gives me the active
WAL, and I can match somehow this:

Latest checkpoint location:           7/F70021A0
Prior checkpoint location:            7/F7002110
Latest checkpoint's REDO location:    7/F7002168
Latest checkpoint's TimeLineID:       2

with this filename: 0000000200000007000000F7

and that's all.

thanks,

--
rd

Peel your own image from the mirror.
Sit. Feast on your life.

On 04/19/2011 05:29 PM, rudi wrote:
> $ psql -c "SELECT pg_start_backup('switchover base backup')"
> $ rsync -av /var/lib/postgresql/9.0/main/ --delete --exclude server.crt
> --exclude server.key --exclude recovery.* --exclude postmaster.pid
> --exclude archive $FORMER_MASTER_ADDRESS:/var/lib/postgresql/9.0/main/
> $ psql -c "SELECT pg_stop_backup()"

No, it doesn't work. The slave is stuck in "archive recovery":

Database cluster state: in archive recovery

and doesn't accept connections:

2011-04-20 10:17:00 CEST:[local]:u@postgres:[13099] FATAL:  the database
system is starting up

even if is a hot standby.

I really can't find a way to do a safe switchover.

--
rd

Peel your own image from the mirror.
Sit. Feast on your life.

On 04/20/2011 10:26 AM, rudi wrote:
> Database cluster state: in archive recovery
>
> and doesn't accept connections:
>
> 2011-04-20 10:17:00 CEST:[local]:u@postgres:[13099] FATAL: the database
> system is starting up
>
> even if is a hot standby.

My fault: it seems I forgot to stop the backup in my previous attempt. I
tried this procedure and now the former master acts as a hot standby as
expected:

On the former master:

a) stop
b) cleanup pg_xlog
c) set archive_mode = off and hot_standby = on in postgresql.conf

On the former slave:

a) set archive_mode = on in postgresql.conf
b) psql -c "SELECT pg_start_backup('switchover base backup')"
c) rsync -av $PGDATA --delete --exclude server.crt --exclude server.key
--exclude recovery.* --exclude postmaster.pid --exclude pg_xlog
--exclude archive $SLAVE:$PGDATA
d) psql -c "SELECT pg_stop_backup()"

Finally I started the former master.

--
rd

Peel your own image from the mirror.
Sit. Feast on your life.