Обсуждение: tar, but not gnu tar
Hi Everybody,
According to section 23.3.2 of 8.2.4 manual:
Also, some versions of GNU tar consider it
an error if a file is changed while tar is
copying it. There does not seem to be any
very convenient way to distinguish this
error from other types of errors, other
than manual inspection of tar's messages.
GNU tar is therefore not the best tool for
making base backups.
On my linux machine, gnu tar is the tar. Does
anybody have a suggestion as to where I can go
to get a tar that is not gnu?
Thanks in advance.
Tena Sakai
tsakai@gallo.ucsf.edu
>>> On Tue, Aug 21, 2007 at 7:28 PM, in message <FE44E0D7EAD2ED4BB2165071DB8E328C03062B59@egcrc-ex01.egcrc.org>, "Tena Sakai" <tsakai@gallo.ucsf.edu> wrote: > > On my linux machine, gnu tar is the tar. Does > anybody have a suggestion as to where I can go > to get a tar that is not gnu? Have you considered using cpio instead? http://www.gnu.org/software/cpio/ -Kevin
Tena Sakai wrote:
> Hi Everybody,
>
> According to section 23.3.2 of 8.2.4 manual:
>
> Also, some versions of GNU tar consider it
> an error if a file is changed while tar is
> copying it. There does not seem to be any
> very convenient way to distinguish this
> error from other types of errors, other
> than manual inspection of tar's messages.
> GNU tar is therefore not the best tool for
> making base backups.
>
> On my linux machine, gnu tar is the tar. Does
> anybody have a suggestion as to where I can go
> to get a tar that is not gnu?
We have updated the 8.3 documentation to be more accurate about GNU tar:
Some backup tools that you might wish to use emit warnings or errors
if the files they are trying to copy change while the copy proceeds.
This situation is normal, and not an error, when taking a base backup
of an active database; so you need to ensure that you can distinguish
complaints of this sort from real errors. For example, some versions
of <application>rsync</> return a separate exit code for
<quote>vanished source files</>, and you can write a driver script to
accept this exit code as a non-error case. Also, some versions of
GNU <application>tar</> consider it an error if a file is changed
while <application>tar</> is copying it. Fortunately, GNU
<application>tar</> versions 1.16 and later exit with <literal>1</>
if files changed during the backup, and <literal>2</> for other errors.
so your version of 'tar' might be fine.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Hi Kevin,
Yes, I have, but I am much more familiar with tar.
I think I will go with the latest gnu tar (v 1.18)
which is suggested by Bruce. I will play with it
tomorrow and see how it goes.
Thanks.
Tena
tsakai@gallo.ucsf.edu
-----Original Message-----
From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov]
Sent: Tue 8/21/2007 6:02 PM
To: Tena Sakai; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] tar, but not gnu tar
>>> On Tue, Aug 21, 2007 at 7:28 PM, in message
<FE44E0D7EAD2ED4BB2165071DB8E328C03062B59@egcrc-ex01.egcrc.org>, "Tena Sakai"
<tsakai@gallo.ucsf.edu> wrote:
>
> On my linux machine, gnu tar is the tar. Does
> anybody have a suggestion as to where I can go
> to get a tar that is not gnu?
Have you considered using cpio instead?
http://www.gnu.org/software/cpio/
-Kevin
Tena, We have been very happy with star. It is a very nice pax, cpio, gnutar,... replacement. You may want to give it a try. Ken On Tue, Aug 21, 2007 at 06:53:53PM -0700, Tena Sakai wrote: > Hi Kevin, > > Yes, I have, but I am much more familiar with tar. > I think I will go with the latest gnu tar (v 1.18) > which is suggested by Bruce. I will play with it > tomorrow and see how it goes. > > Thanks. > > Tena > > tsakai@gallo.ucsf.edu > > > -----Original Message----- > From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov] > Sent: Tue 8/21/2007 6:02 PM > To: Tena Sakai; pgsql-admin@postgresql.org > Subject: Re: [ADMIN] tar, but not gnu tar > > >>> On Tue, Aug 21, 2007 at 7:28 PM, in message > <FE44E0D7EAD2ED4BB2165071DB8E328C03062B59@egcrc-ex01.egcrc.org>, "Tena Sakai" > <tsakai@gallo.ucsf.edu> wrote: > > > > On my linux machine, gnu tar is the tar. Does > > anybody have a suggestion as to where I can go > > to get a tar that is not gnu? > > Have you considered using cpio instead? > > http://www.gnu.org/software/cpio/ > > -Kevin > > > >
Thanks, Ken.
I just glanced at man page for star. It looks promising
and I will experiment with it. This may be the ticket.
Regards,
Tena
tsakai@gallo.ucsf.edu
-----Original Message-----
From: Kenneth Marshall [mailto:ktm@rice.edu]
Sent: Wed 8/22/2007 5:30 AM
To: Tena Sakai
Cc: Kevin Grittner; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] tar, but not gnu tar
Tena,
We have been very happy with star. It is a very nice pax,
cpio, gnutar,... replacement. You may want to give it a try.
Ken
On Tue, Aug 21, 2007 at 06:53:53PM -0700, Tena Sakai wrote:
> Hi Kevin,
>
> Yes, I have, but I am much more familiar with tar.
> I think I will go with the latest gnu tar (v 1.18)
> which is suggested by Bruce. I will play with it
> tomorrow and see how it goes.
>
> Thanks.
>
> Tena
>
> tsakai@gallo.ucsf.edu
>
>
> -----Original Message-----
> From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov]
> Sent: Tue 8/21/2007 6:02 PM
> To: Tena Sakai; pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] tar, but not gnu tar
>
> >>> On Tue, Aug 21, 2007 at 7:28 PM, in message
> <FE44E0D7EAD2ED4BB2165071DB8E328C03062B59@egcrc-ex01.egcrc.org>, "Tena Sakai"
> <tsakai@gallo.ucsf.edu> wrote:
> >
> > On my linux machine, gnu tar is the tar. Does
> > anybody have a suggestion as to where I can go
> > to get a tar that is not gnu?
>
> Have you considered using cpio instead?
>
> http://www.gnu.org/software/cpio/
>
> -Kevin
>
>
>
>
Hi Everybody,
I had a bit of time to experiment with tar and star today
and I am no longer sure what the real issue is. Perhaps
some of you can clarify. Here's the test I ran:
There are many a ways to get the same thing done, but I
did it in a most intuitive way (to me). I had 3 windows
to the same directory. In one window, I ran a simple
shell program interactively and continuously:
while true
do
touch big.inputfile
done
And in the 2nd window, I ran a command:
tar cf ../moo.tar . > tar_errlog 2>&1
There are a bunch of files in this directory and below,
but one of them (big.inputfile) is 4.7gb. The above
tar command exited with exit status 0 and a line like:
tar: ./input.big: file changed as we read it
The tar I used is version 1.14.
Also, in the 3rd window, I ran a command:
star cf ../moo.star . > star_errlog 2>&1
Everything else remained the same. Here, start exited
with 0 as well, while it gave me this line in errlog:
star: 418759 blocks + 0 bytes (total of 4288092160 bytes = 4187590.00k).
The star version is 1.5a25.
Here's what started this very thread:
> Some backup tools that you might wish to use emit warnings or errors
> if the files they are trying to copy change while the copy proceeds.
> This situation is normal, and not an error, when taking a base backup
> of an active database; so you need to ensure that you can distinguish
> complaints of this sort from real errors. For example, some versions
> of rsync return a separate exit code for "vanished source files", and
> you can write a driver script to accept this exit code as a non-error
> case. Also, some versions of GNU tar consider it an error if a file is
> changed while tar is copying it. There does not seem to be any very
> convenient way to distinguish this error from other types of errors,
> other than manual inspection of tar's messages. GNU tar is therefore
> not the best tool for making base backups.
Is the issue to do with the exit code of the backup program? If that's
the case, both tar and star (at least with these very versions) are not
acceptable. Or is the issue to do with what the program spits out as
human readable message, which should be or shouldn't be ignored?
Another big problem I see is that neither program lists the exit codes
in the man pages. That's not good. Rsync man page has a section where
all exit code is listed, but I have no foreign drive on this host. I
am tempted to get a tiny machine just to do the backup.
I'd be interested your insights and thoughts.
Regards,
Tena
-----Original Message-----
From: Kenneth Marshall [mailto:ktm@rice.edu]
Sent: Wed 8/22/2007 5:30 AM
To: Tena Sakai
Cc: Kevin Grittner; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] tar, but not gnu tar
Tena,
We have been very happy with star. It is a very nice pax,
cpio, gnutar,... replacement. You may want to give it a try.
Ken
On Tue, Aug 21, 2007 at 06:53:53PM -0700, Tena Sakai wrote:
> Hi Kevin,
>
> Yes, I have, but I am much more familiar with tar.
> I think I will go with the latest gnu tar (v 1.18)
> which is suggested by Bruce. I will play with it
> tomorrow and see how it goes.
>
> Thanks.
>
> Tena
>
> tsakai@gallo.ucsf.edu
>
>
> -----Original Message-----
> From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov]
> Sent: Tue 8/21/2007 6:02 PM
> To: Tena Sakai; pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] tar, but not gnu tar
>
> >>> On Tue, Aug 21, 2007 at 7:28 PM, in message
> <FE44E0D7EAD2ED4BB2165071DB8E328C03062B59@egcrc-ex01.egcrc.org>, "Tena Sakai"
> <tsakai@gallo.ucsf.edu> wrote:
> >
> > On my linux machine, gnu tar is the tar. Does
> > anybody have a suggestion as to where I can go
> > to get a tar that is not gnu?
>
> Have you considered using cpio instead?
>
> http://www.gnu.org/software/cpio/
>
> -Kevin
>
>
>
>
Tena Sakai wrote: > Hi Everybody, > > I had a bit of time to experiment with tar and star today > and I am no longer sure what the real issue is. Perhaps > some of you can clarify. Here's the test I ran: > > There are many a ways to get the same thing done, but I > did it in a most intuitive way (to me). I had 3 windows > to the same directory. In one window, I ran a simple > shell program interactively and continuously: > while true > do > touch big.inputfile > done I don't think 'touch' is enough for tar to see the file as changed (you are only updating metadata). (tar did complain but the file contents didn't so it is hard to say if that is a good test.) You should change the file contents during the backup. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce Momjian <bruce@momjian.us> writes:
> Tena Sakai wrote:
>> I had a bit of time to experiment with tar and star today
>> and I am no longer sure what the real issue is. Perhaps
>> some of you can clarify. Here's the test I ran:
> I don't think 'touch' is enough for tar to see the file as changed (you
> are only updating metadata). (tar did complain but the file contents
> didn't so it is hard to say if that is a good test.) You should change
> the file contents during the backup.
In fact, I'll bet that you have to change the file *length* during the
backup to trigger gnu tar's complaint. If it were rigorously checking
for file content change, it'd have to read the whole of every file
twice, which hardly seems like overhead that anyone would accept. But a
check for length change would just mean one extra stat() call per file,
which is a whole lot more plausible.
regards, tom lane
Hi Tom,
Hi Bruce,
Thanks for your input. Fair enough. I redid it.
This time, in the first window, I ran this mini
shell program continuously:
while true
> do
> date >> big.input
> done
In the 2nd window:
tar cf ../moo.tar . > tar_errlog 2>&1
Tar returned the exit code 0. In the tar_errlog, it said:
tar: ./big.input: file changed as we read it
In the 3rd:
star cf ../moo.star . > star_errlog 2>&1
Star returned exit code of 254. 6 lines were in
star_errlog:
star: 'big.input': file changed size (increased).
star: 419190 blocks + 0 bytes (total of 4292505600 bytes = 4191900.00k).
star: The following problems occurred during archive processing:
star: Cannot: stat 0, open 0, read/write 0. Size changed 1.
star: Missing links 0, Name too long 0, File too big 0, Not dumped 0.
star: Processed all possible files, despite earlier errors.
That was so different from the previous test, I ran
star one more time, but this time I killed the process
in the first window. Ie., I ran the same star test while
nothing was changing. It ruturned exit code 0, and a line
in the star_errlog:
star: 419467 blocks + 0 bytes (total of 4295342080 bytes = 4194670.00k).
I am inclined to use star. At the bottom of the star man page,
the author Joerg Schilling gives out his email address (3 of
them!) and I will email him and ask if he can supply a list of
all exit codes.
Regards,
Tena Sakai
tsakai@gallo.ucsf.edu
-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Wed 8/22/2007 10:18 PM
To: Bruce Momjian
Cc: Tena Sakai; Kenneth Marshall; Kevin Grittner; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] tar, but not gnu tar
Bruce Momjian <bruce@momjian.us> writes:
> Tena Sakai wrote:
>> I had a bit of time to experiment with tar and star today
>> and I am no longer sure what the real issue is. Perhaps
>> some of you can clarify. Here's the test I ran:
> I don't think 'touch' is enough for tar to see the file as changed (you
> are only updating metadata). (tar did complain but the file contents
> didn't so it is hard to say if that is a good test.) You should change
> the file contents during the backup.
In fact, I'll bet that you have to change the file *length* during the
backup to trigger gnu tar's complaint. If it were rigorously checking
for file content change, it'd have to read the whole of every file
twice, which hardly seems like overhead that anyone would accept. But a
check for length change would just mean one extra stat() call per file,
which is a whole lot more plausible.
regards, tom lane
Tena Sakai wrote: > Hi Tom, > Hi Bruce, > > Thanks for your input. Fair enough. I redid it. > This time, in the first window, I ran this mini > shell program continuously: > > while true > > do > > date >> big.input > > done > > In the 2nd window: > > tar cf ../moo.tar . > tar_errlog 2>&1 > > Tar returned the exit code 0. In the tar_errlog, it said: > tar: ./big.input: file changed as we read it This seems to contradict what we say about GNU tar? Is this GNU tar? What version? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Hi Bruce,
> This seems to contradict what we say about GNU tar?
> Is this GNU tar? What version?
Yes, it is GNU tar v1.14
Regards,
Tena Sakai
tsakai@gallo.ucsf.edu
-----Original Message-----
From: Bruce Momjian [mailto:bruce@momjian.us]
Sent: Thu 8/23/2007 12:02 PM
To: Tena Sakai
Cc: Tom Lane; Kenneth Marshall; Kevin Grittner; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] tar, but not gnu tar
Tena Sakai wrote:
> Hi Tom,
> Hi Bruce,
>
> Thanks for your input. Fair enough. I redid it.
> This time, in the first window, I ran this mini
> shell program continuously:
>
> while true
> > do
> > date >> big.input
> > done
>
> In the 2nd window:
>
> tar cf ../moo.tar . > tar_errlog 2>&1
>
> Tar returned the exit code 0. In the tar_errlog, it said:
> tar: ./big.input: file changed as we read it
This seems to contradict what we say about GNU tar? Is this GNU tar?
What version?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
"Tena Sakai" <tsakai@gallo.ucsf.edu> writes:
>> This seems to contradict what we say about GNU tar?
>> Is this GNU tar? What version?
> Yes, it is GNU tar v1.14
FWIW, I tried this on Fedora Core 6 while running pgbench:
[tgl@rh2 ~]$ tar --version
tar (GNU tar) 1.15.1
[tgl@rh2 ~]$ tar cf t.tar $PGDATA
tar: Removing leading `/' from member names
tar: /home/tgl/testversion/data/pg_xlog/000000010000000000000003: file changed as we read it
[tgl@rh2 ~]$ echo $?
0
[tgl@rh2 ~]$
ISTR that the original caution was against writing scripts that assume
anything being emitted to stderr must indicate a problem.
regards, tom lane
Am Freitag, 24. August 2007 03:58 schrieb Tom Lane: > "Tena Sakai" <tsakai@gallo.ucsf.edu> writes: > >> This seems to contradict what we say about GNU tar? > >> Is this GNU tar? What version? > > > > Yes, it is GNU tar v1.14 > > FWIW, I tried this on Fedora Core 6 while running pgbench: > > [tgl@rh2 ~]$ tar --version > tar (GNU tar) 1.15.1 > [tgl@rh2 ~]$ tar cf t.tar $PGDATA > tar: Removing leading `/' from member names > tar: /home/tgl/testversion/data/pg_xlog/000000010000000000000003: file > changed as we read it [tgl@rh2 ~]$ echo $? > 0 > [tgl@rh2 ~]$ > > ISTR that the original caution was against writing scripts that assume > anything being emitted to stderr must indicate a problem. The relevant NEWS entry from GNU tar 1.16 is: """ * After creating an archive, tar exits with code 1 if some files were changed while being read. Previous versions exited with code 2 (fatal error), and only if some files were truncated while being archived. """ -- Peter Eisentraut http://developer.postgresql.org/~petere/
Peter Eisentraut wrote:
> Am Freitag, 24. August 2007 03:58 schrieb Tom Lane:
> > "Tena Sakai" <tsakai@gallo.ucsf.edu> writes:
> > >> This seems to contradict what we say about GNU tar?
> > >> Is this GNU tar? What version?
> > >
> > > Yes, it is GNU tar v1.14
> >
> > FWIW, I tried this on Fedora Core 6 while running pgbench:
> >
> > [tgl@rh2 ~]$ tar --version
> > tar (GNU tar) 1.15.1
> > [tgl@rh2 ~]$ tar cf t.tar $PGDATA
> > tar: Removing leading `/' from member names
> > tar: /home/tgl/testversion/data/pg_xlog/000000010000000000000003: file
> > changed as we read it [tgl@rh2 ~]$ echo $?
> > 0
> > [tgl@rh2 ~]$
> >
> > ISTR that the original caution was against writing scripts that assume
> > anything being emitted to stderr must indicate a problem.
>
> The relevant NEWS entry from GNU tar 1.16 is:
>
> """
> * After creating an archive, tar exits with code 1 if some files were
> changed while being read. Previous versions exited with code 2 (fatal
> error), and only if some files were truncated while being archived.
Docs updated and backpatched to 8.2.X:
Also, some versions of GNU <application>tar</> consider it an error
if a file was truncated while <application>tar</> is copying it.
Fortunately, GNU <application>tar</> versions 1.16 and later exits
with <literal>1</> if a file was changed during the backup, and
<literal>2</> for other errors.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
bruce wrote:
> > > ISTR that the original caution was against writing scripts that assume
> > > anything being emitted to stderr must indicate a problem.
> >
> > The relevant NEWS entry from GNU tar 1.16 is:
> >
> > """
> > * After creating an archive, tar exits with code 1 if some files were
> > changed while being read. Previous versions exited with code 2 (fatal
> > error), and only if some files were truncated while being archived.
>
> Docs updated and backpatched to 8.2.X:
>
> Also, some versions of GNU <application>tar</> consider it an error
> if a file was truncated while <application>tar</> is copying it.
> Fortunately, GNU <application>tar</> versions 1.16 and later exits
> with <literal>1</> if a file was changed during the backup, and
> <literal>2</> for other errors.
I have updated the documentation to mention that the real problem is an
"indistinguishable" error return code:
Also, some versions of GNU <application>tar</> return an error code
indistinguishable from a fatal error if a file was truncated while
<application>tar</> was copying it. Fortunately, GNU
<application>tar</> versions 1.16 and later exits with <literal>1</>
if a file was changed during the backup, and <literal>2</> for other
errors.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +