Обсуждение: [GENERAL] pg_basebackup ----xlog-method=stream
I used this command to set up a streaming replica and it worked perfectly.
I tried to run it to create an online backup of the master on that replica for backup purposes and it seems not to have worked as well.pg_basebackup: could not get transaction log end position from server: ERROR: requested WAL segment 000000010000496F000000BF has already been removed
# cat 000000010000496F000000BF.00578DF8.backup
START WAL LOCATION: 496F/BF578DF8 (file 000000010000496F000000BF)
STOP WAL LOCATION: 4971/11FC6528 (file 000000010000497100000011)
CHECKPOINT LOCATION: 496F/C1ACE0D8
BACKUP METHOD: streamed
BACKUP FROM: master
START TIME: 2017-04-21 01:08:38 PDT
LABEL: full-20170421
STOP TIME: 2017-04-21 06:31:28 PDT
The files were definitely being streamed, I restarted and on the master I see this:
[root@db20 ~]# ps aux | grep stream
postgres 113855 0.4 0.0 274955352 2932 ? Ss 10:53 0:00 postgres: wal sender process postgres 192.168.4.21(51292) streaming 4972/7FBC22B8
root 115958 0.0 0.0 112652 956 pts/2 S+ 10:56 0:00 grep --color=auto stream
postgres 142072 0.3 0.0 274955348 3160 ? Ss Apr20 3:47 postgres: wal sender process postgres 192.168.4.21(51240) streaming 4972/7FBC22B8
[root@db20 ~]# ps aux | grep stream
postgres 113855 0.4 0.0 274955352 2932 ? Ss 10:53 0:00 postgres: wal sender process postgres 192.168.4.21(51292) streaming 4972/7FBC22B8
root 115958 0.0 0.0 112652 956 pts/2 S+ 10:56 0:00 grep --color=auto stream
postgres 142072 0.3 0.0 274955348 3160 ? Ss Apr20 3:47 postgres: wal sender process postgres 192.168.4.21(51240) streaming 4972/7FBC22B8
although the --verbose output from pg_basebackup mentions starting the streaming, it doesn't mention the file names like it does for the datadir files.
On 04/21/2017 11:02 AM, Ian Harding wrote: > I used this command to set up a streaming replica and it worked perfectly. Was that the full command? If not can you show the full command you used? Postgres version? > > I tried to run it to create an online backup of the master on that > replica for backup purposes and it seems not to have worked as well. > > I thought that streaming the WAL would eliminate the need to keep tons > of WAL around, that since they were streamed concurrently with the file > they would be in place. However, I got > > pg_basebackup: could not get transaction log end position from server: > ERROR: requested WAL segment 000000010000496F000000BF has already been > removed Where was pg_basebackup running, on the master or somewhere else? The pg_basebackup was from the same Postgres version as the master? > > The ...backup file in the archive logs folder of the master looks like this" > > # cat 000000010000496F000000BF.00578DF8.backup > START WAL LOCATION: 496F/BF578DF8 (file 000000010000496F000000BF) > STOP WAL LOCATION: 4971/11FC6528 (file 000000010000497100000011) > CHECKPOINT LOCATION: 496F/C1ACE0D8 > BACKUP METHOD: streamed > BACKUP FROM: master > START TIME: 2017-04-21 01:08:38 PDT > LABEL: full-20170421 > STOP TIME: 2017-04-21 06:31:28 PDT > > so the "...already been removed" message is referring to the very first > file. > > The files were definitely being streamed, I restarted and on the master > I see this: > > [root@db20 ~]# ps aux | grep stream > postgres 113855 0.4 0.0 274955352 2932 ? Ss 10:53 0:00 > postgres: wal sender process postgres 192.168.4.21(51292) streaming > 4972/7FBC22B8 > root 115958 0.0 0.0 112652 956 pts/2 S+ 10:56 0:00 grep > --color=auto stream > postgres 142072 0.3 0.0 274955348 3160 ? Ss Apr20 3:47 > postgres: wal sender process postgres 192.168.4.21(51240) streaming > 4972/7FBC22B8 > > > although the --verbose output from pg_basebackup mentions starting the > streaming, it doesn't mention the file names like it does for the > datadir files. > > Am I misunderstanding how this works? I have WAL archiving set up, so > the files are available, but I wanted them included in the backup. > > Thanks! > > - Ian -- Adrian Klaver adrian.klaver@aklaver.com
On Sat, Apr 22, 2017 at 3:02 AM, Ian Harding <harding.ian@gmail.com> wrote: > Am I misunderstanding how this works? I have WAL archiving set up, so the > files are available, but I wanted them included in the backup. Please note that if you have a WAL archive available, you may not even need to have --xlog-method=stream, which is useful to have fully safe-contailed backups. For the error regarding the missing WAL segment, it could happen if Postgres completes two checkpoints when taking the backup. One thing that you could use is a replication slot that gives the guarantee that segments are retained. Those have been integrated in pg_basebackup with 9.6. -- Michael
> On Apr 21, 2017, at 3:29 PM, Michael Paquier <michael.paquier@gmail.com> wrote: > >> On Sat, Apr 22, 2017 at 3:02 AM, Ian Harding <harding.ian@gmail.com> wrote: >> Am I misunderstanding how this works? I have WAL archiving set up, so the >> files are available, but I wanted them included in the backup. > > Please note that if you have a WAL archive available, you may not even > need to have --xlog-method=stream, which is useful to have fully > safe-contailed backups. For the error regarding the missing WAL > segment, it could happen if Postgres completes two checkpoints when > taking the backup. One thing that you could use is a replication slot > that gives the guarantee that segments are retained. Those have been > integrated in pg_basebackup with 9.6. Yeah. I really want to use replication slots but put the server into production without setting the appropriate config settings.I need to schedule some downtime for a restart. > -- > Michael