Обсуждение: pgsql: Allow pg_basebackup to stream transaction log in tar mode
Allow pg_basebackup to stream transaction log in tar mode This will write the received transaction log into a file called pg_wal.tar(.gz) next to the other tarfiles instead of writing it to base.tar. When using fetch mode, the transaction log is still written to base.tar like before, and when used against a pre-10 server, the file is named pg_xlog.tar. To do this, implement a new concept of a "walmethod", which is responsible for writing the WAL. Two implementations exist, one that writes to a plain directory (which is also used by pg_receivexlog) and one that writes to a tar file with optional compression. Reviewed by Michael Paquier Branch ------ master Details ------- http://git.postgresql.org/pg/commitdiff/56c7d8d4552180fd66fe48423bb2a9bb767c2d87 Modified Files -------------- doc/src/sgml/ref/pg_basebackup.sgml | 18 +- src/bin/pg_basebackup/Makefile | 2 +- src/bin/pg_basebackup/pg_basebackup.c | 62 +- src/bin/pg_basebackup/pg_receivexlog.c | 10 +- src/bin/pg_basebackup/receivelog.c | 316 ++++------ src/bin/pg_basebackup/receivelog.h | 3 +- src/bin/pg_basebackup/t/010_pg_basebackup.pl | 6 +- src/bin/pg_basebackup/walmethods.c | 886 +++++++++++++++++++++++++++ src/bin/pg_basebackup/walmethods.h | 45 ++ src/include/pgtar.h | 1 + src/port/tar.c | 2 +- 11 files changed, 1107 insertions(+), 244 deletions(-)
That broke some buildfarm. Looking into it.
//Magnus
On Sun, Oct 23, 2016 at 3:28 PM, Magnus Hagander <magnus@hagander.net> wrote:
Allow pg_basebackup to stream transaction log in tar mode
This will write the received transaction log into a file called
pg_wal.tar(.gz) next to the other tarfiles instead of writing it to
base.tar. When using fetch mode, the transaction log is still written to
base.tar like before, and when used against a pre-10 server, the file
is named pg_xlog.tar.
To do this, implement a new concept of a "walmethod", which is
responsible for writing the WAL. Two implementations exist, one that
writes to a plain directory (which is also used by pg_receivexlog) and
one that writes to a tar file with optional compression.
Reviewed by Michael Paquier
Branch
------
master
Details
-------
http://git.postgresql.org/pg/commitdiff/ 56c7d8d4552180fd66fe48423bb2a9 bb767c2d87
Modified Files
--------------
doc/src/sgml/ref/pg_basebackup.sgml | 18 +-
src/bin/pg_basebackup/Makefile | 2 +-
src/bin/pg_basebackup/pg_basebackup.c | 62 +-
src/bin/pg_basebackup/pg_receivexlog.c | 10 +-
src/bin/pg_basebackup/receivelog.c | 316 ++++------
src/bin/pg_basebackup/receivelog.h | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 6 +-
src/bin/pg_basebackup/walmethods.c | 886 +++++++++++++++++++++++++++
src/bin/pg_basebackup/walmethods.h | 45 ++
src/include/pgtar.h | 1 +
src/port/tar.c | 2 +-
11 files changed, 1107 insertions(+), 244 deletions(-)
--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers
The remaining windows buildfarm build issue looks like this:
src/bin/pg_basebackup/receivelog.c(135): error C2039: '_commit' : is not a member of 'WalWriteMethod' [C:\buildfarm\buildenv\HEAD\pgsql.build\pg_basebackup.vcxproj]
AFAICT this comes from using walmethod->fsync(), and we have a #define changing fsync() to _commit (port/win32.h, line 70).
The easiest might just be to rename walmethod->fsync() to walmethod->do_fsync(), so we don't have to mess with any other code.
Thoughts?
//Magnus
On Sun, Oct 23, 2016 at 3:55 PM, Magnus Hagander <magnus@hagander.net> wrote:
That broke some buildfarm. Looking into it.//MagnusOn Sun, Oct 23, 2016 at 3:28 PM, Magnus Hagander <magnus@hagander.net> wrote:Allow pg_basebackup to stream transaction log in tar mode
This will write the received transaction log into a file called
pg_wal.tar(.gz) next to the other tarfiles instead of writing it to
base.tar. When using fetch mode, the transaction log is still written to
base.tar like before, and when used against a pre-10 server, the file
is named pg_xlog.tar.
To do this, implement a new concept of a "walmethod", which is
responsible for writing the WAL. Two implementations exist, one that
writes to a plain directory (which is also used by pg_receivexlog) and
one that writes to a tar file with optional compression.
Reviewed by Michael Paquier
Branch
------
master
Details
-------
http://git.postgresql.org/pg/commitdiff/56c7d8d4552180fd66fe 48423bb2a9bb767c2d87
Modified Files
--------------
doc/src/sgml/ref/pg_basebackup.sgml | 18 +-
src/bin/pg_basebackup/Makefile | 2 +-
src/bin/pg_basebackup/pg_basebackup.c | 62 +-
src/bin/pg_basebackup/pg_receivexlog.c | 10 +-
src/bin/pg_basebackup/receivelog.c | 316 ++++------
src/bin/pg_basebackup/receivelog.h | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 6 +-
src/bin/pg_basebackup/walmethods.c | 886 +++++++++++++++++++++++++++
src/bin/pg_basebackup/walmethods.h | 45 ++
src/include/pgtar.h | 1 +
src/port/tar.c | 2 +-
11 files changed, 1107 insertions(+), 244 deletions(-)
--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers
Magnus Hagander <magnus@hagander.net> writes:
> The remaining windows buildfarm build issue looks like this:
> src/bin/pg_basebackup/receivelog.c(135): error C2039: '_commit' : is
> not a member of 'WalWriteMethod'
> [C:\buildfarm\buildenv\HEAD\pgsql.build\pg_basebackup.vcxproj]
> AFAICT this comes from using walmethod->fsync(), and we have a #define
> changing fsync() to _commit (port/win32.h, line 70).
Yeah.
> The easiest might just be to rename walmethod->fsync() to
> walmethod->do_fsync(), so we don't have to mess with any other code.
How about calling the method just "sync()"? Seems a bit less
artificial.
regards, tom lane
On Sun, Oct 23, 2016 at 5:48 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Magnus Hagander <magnus@hagander.net> writes:
> The remaining windows buildfarm build issue looks like this:
> src/bin/pg_basebackup/receivelog.c(135): error C2039: '_commit' : is
> not a member of 'WalWriteMethod'
> [C:\buildfarm\buildenv\HEAD\pgsql.build\pg_basebackup. vcxproj]
> AFAICT this comes from using walmethod->fsync(), and we have a #define
> changing fsync() to _commit (port/win32.h, line 70).
Yeah.
> The easiest might just be to rename walmethod->fsync() to
> walmethod->do_fsync(), so we don't have to mess with any other code.
How about calling the method just "sync()"? Seems a bit less
artificial.
Sure, that should work too. Seems we're not defining that one to something else, so let's use that.
I'll go fix.
I have following warning while compiling with clang.
walmethods.c:437:9: warning: comparison of unsigned expression < 0 is always false [-Wtautological-compare]
if (r < 0)
~ ^ ~
1 warning generated.
It seems that 'r' should be 'ssize_t' instead of 'size_t'.
------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
On Sun, Oct 23, 2016 at 4:28 PM, Magnus Hagander <magnus@hagander.net> wrote:
Allow pg_basebackup to stream transaction log in tar mode
This will write the received transaction log into a file called
pg_wal.tar(.gz) next to the other tarfiles instead of writing it to
base.tar. When using fetch mode, the transaction log is still written to
base.tar like before, and when used against a pre-10 server, the file
is named pg_xlog.tar.
To do this, implement a new concept of a "walmethod", which is
responsible for writing the WAL. Two implementations exist, one that
writes to a plain directory (which is also used by pg_receivexlog) and
one that writes to a tar file with optional compression.
Reviewed by Michael Paquier
Branch
------
master
Details
-------
http://git.postgresql.org/pg/commitdiff/ 56c7d8d4552180fd66fe48423bb2a9 bb767c2d87
Modified Files
--------------
doc/src/sgml/ref/pg_basebackup.sgml | 18 +-
src/bin/pg_basebackup/Makefile | 2 +-
src/bin/pg_basebackup/pg_basebackup.c | 62 +-
src/bin/pg_basebackup/pg_receivexlog.c | 10 +-
src/bin/pg_basebackup/receivelog.c | 316 ++++------
src/bin/pg_basebackup/receivelog.h | 3 +-
src/bin/pg_basebackup/t/010_pg_basebackup.pl | 6 +-
src/bin/pg_basebackup/walmethods.c | 886 +++++++++++++++++++++++++++
src/bin/pg_basebackup/walmethods.h | 45 ++
src/include/pgtar.h | 1 +
src/port/tar.c | 2 +-
11 files changed, 1107 insertions(+), 244 deletions(-)
--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers
On Mon, Oct 24, 2016 at 2:58 PM, Alexander Korotkov <a.korotkov@postgrespro.ru> wrote:
I have following warning while compiling with clang.walmethods.c:437:9: warning: comparison of unsigned expression < 0 is always false [-Wtautological-compare]if (r < 0)~ ^ ~1 warning generated.It seems that 'r' should be 'ssize_t' instead of 'size_t'.
Yup. The result from tar_write() is ssize_t so it definitely should be.
Thanks, I've applied a fix.
//Magnus