Обсуждение: Tape/DVD Backup Suggestions?
I'm looking into new ways of backing up the data in our lab, including the PostgreSQL database. Currently, we have a single DDS-2 tape drive capable of holding 8Gig compressed. However, it is slow (i.e. takes a day to backup about 40 Gig of information), spans multiple tapes, and makes it hard to find just a single file or two on the backup (i.e. I have to go through many tapes before I can find and extract one of two files). Our CDRW backups are easier to manage and relatively fast, but require dozens of CDRWs. Can anyone make suggestions on backup systems? I was thinking that some sort of DVD writing system would be good for accessing one or two files in the backup quickly. It would probably also complete backups faster. However, I think DVD's only hold a few Gigs on them. Are there systems like with these features that could handle say 20 Gigs per media? Thanks. -Tony
> Can anyone make suggestions on backup systems? I was thinking that
> some sort of DVD writing system would be good for accessing one or two
> files in the backup quickly. It would probably also complete backups
> faster. However, I think DVD's only hold a few Gigs on them. Are there
> systems like with these features that could handle say 20 Gigs per
> media?
Where I work we just started using a new disk based backup system that
we wrote in-house. Disk is a little bit cheaper then tape -- plus much
faster. Another bonus is that your compression rate does not depend on
how fast your data is flowing. I'd suggest you buy a few 160GB IDE
drives and just dump your data there. It's faster, easier and you don't
need to change tapes. If you're interested in the source to our system
just let me know.
--Nate
Nathan Mueller wrote:
>>Can anyone make suggestions on backup systems? I was thinking that
>>some sort of DVD writing system would be good for accessing one or two
>>files in the backup quickly. It would probably also complete backups
>>faster. However, I think DVD's only hold a few Gigs on them. Are there
>>systems like with these features that could handle say 20 Gigs per
>>media?
>
>
> Where I work we just started using a new disk based backup system that
> we wrote in-house. Disk is a little bit cheaper then tape -- plus much
> faster. Another bonus is that your compression rate does not depend on
> how fast your data is flowing. I'd suggest you buy a few 160GB IDE
> drives and just dump your data there. It's faster, easier and you don't
> need to change tapes. If you're interested in the source to our system
> just let me know.
I'd second that. Take a look at rdiff-backup:
http://rdiff-backup.stanford.edu/
It can backup remotely, and keeps a complete up-to-date backup as well
as reverse diffs so you can do point-in-time restore.
HTH,
Joe
On Wed, Jul 17, 2002 at 04:48:10PM -0500, Nathan Mueller wrote: > > Are there > > systems like with these features that could handle say 20 Gigs per > > media? > > Where I work we just started using a new disk based backup system that > we wrote in-house. Disk is a little bit cheaper then tape -- plus much > faster. Another bonus is that your compression rate does not depend on > how fast your data is flowing. I'd suggest you buy a few 160GB IDE > drives and just dump your data there. It's faster, easier and you don't > need to change tapes. If you're interested in the source to our system > just let me know. I advise against hard disk based backups: It's not cheap, sure, a tape drive is expensive, but the tapes are quite cheap. The cheapness of the tapes allows you to use several for different backup strategies (weekly and daily incremental and monthly full backup, for example) and to keep older data on saved tapes. It's easier. There are programs like Arkeia (free for one linux server and two clients (win32/linux)), that makes tape and backup management a few clicks (but a read of documentation is still needed). As a side note, Arkeia supports direct dumping and backup from serveral rdbms, postgresql included. It's more reliable. If the backup disk fails, all backup is lost and its substitution an hassle. If a tape breaks you still have the other tapes for last week/day/etc.. Still, a tape drive can fail or a tape can screw the drive, but I haven't heard of anyone to whom that has happened. In hardware world, quality normally comes with an higher price. That being said, I don't have a tape. Too much for my pocket. :) So I made a script that creates, compress, splits and burns backups to cds. I'll made a switch to dvds when the price for dvd recorders drops to EUR 100 and dvd blank discs to EUR 2... Your solution is still a valid one, but more a special case for those without too much $$... Regards, Luciano Rocha -- Consciousness: that annoying time between naps.
At 01:20 PM 7/17/02 , Tony Reina wrote:
>I was thinking that some sort of DVD writing system would be good for
>accessing one or two files in the backup quickly. It would probably also
>complete backups faster. However, I think DVD's only hold a few Gigs on
>them. Are there systems like with these features that could handle say 20
>Gigs per media?
IIRC a DVD in full goose mode can hold over 7Gbytes. Since databases
typically compress very well, a DVD-R with client-side compression might
do it for you.
-crl
--
Chad R. Larson (CRL22) chad@eldocomp.com
Eldorado Computing, Inc. 602-604-3100
5353 North 16th Street, Suite 400
Phoenix, Arizona 85016-3228
Yes, I'm a little wary of hard disk based systems as my sole backup. I prefer something that allows me to have at least 2-3 different media backups (e.g. one this week, one last week, and one the week before last). Also, I like to be able to take a copy of the media home just because I'm paranoid that the lab will burn down or something silly like that ;>) I've been looking at the Exabyte systems which hold 80G/160G and write at about 10G/hour. Anyone have experience with these? How onerous is it to look at the table of contents or restore a specific file from these tapes? -Tony At 11:35 PM 7/17/02 +0100, strange@nsk.yi.org wrote: >On Wed, Jul 17, 2002 at 04:48:10PM -0500, Nathan Mueller wrote: > > > Are there > > > systems like with these features that could handle say 20 Gigs per > > > media? > > > > Where I work we just started using a new disk based backup system that > > we wrote in-house. Disk is a little bit cheaper then tape -- plus much > > faster. Another bonus is that your compression rate does not depend on > > how fast your data is flowing. I'd suggest you buy a few 160GB IDE > > drives and just dump your data there. It's faster, easier and you don't > > need to change tapes. If you're interested in the source to our system > > just let me know. > >I advise against hard disk based backups: > >It's not cheap, sure, a tape drive is expensive, but the tapes are quite >cheap. > >The cheapness of the tapes allows you to use several for different >backup strategies (weekly and daily incremental and monthly full backup, for >example) and to keep older data on saved tapes. > >It's easier. There are programs like Arkeia (free for one linux server and >two clients (win32/linux)), that makes tape and backup management a few >clicks (but a read of documentation is still needed). As a side note, >Arkeia supports direct dumping and backup from serveral rdbms, postgresql >included. > >It's more reliable. If the backup disk fails, all backup is lost and its >substitution an hassle. If a tape breaks you still have the other tapes >for last week/day/etc.. Still, a tape drive can fail or a tape can screw >the drive, but I haven't heard of anyone to whom that has happened. In >hardware world, quality normally comes with an higher price. > >That being said, I don't have a tape. Too much for my pocket. :) >So I made a script that creates, compress, splits and burns backups to >cds. I'll made a switch to dvds when the price for dvd recorders drops to >EUR 100 and dvd blank discs to EUR 2... > >Your solution is still a valid one, but more a special case for those >without too much $$... > >Regards, >Luciano Rocha > >-- >Consciousness: that annoying time between naps.
On Wednesday, July 17, 2002, at 03:54 PM, Tony Reina wrote: > Yes, I'm a little wary of hard disk based systems as my sole backup. I > prefer something that allows me to have at least 2-3 different media > backups (e.g. one this week, one last week, and one the week before > last). Also, I like to be able to take a copy of the media home just > because I'm paranoid that the lab will burn down or something silly > like that ;>) > > I've been looking at the Exabyte systems which hold 80G/160G and write > at about 10G/hour. Anyone have experience with these? How onerous is > it to look at the table of contents or restore a specific file from > these tapes? > [Disclaimer: I don't know what kind of drive the Exabyte is, but I'm guessing it's some proprietary technology.] My current thoughts on backups are to stay away from anything proprietary. With standard stuff like DDSx, you can buy the drives from different vendors (HP, LaCie, probably more) and the tapes from different vendors (Sony, Verbatim, 3M, etc). The tapes aren't "exotic" in any way, so you can find them at Office Depot in a pinch. Just look at DDS1. How long has it been around? 10+ years? I like that kind of lifespan. If you get proprietary stuff, you (and your data) are at the vendor's mercy. If they go belly-up or decide your product isn't worth supporting anymore, tough luck. --Jeremy
> I advise against hard disk based backups:
Don't write off disk backups yet. I would have (and did) a year ago,
but in the last six months things have gotten a lot cheaper.
> It's not cheap, sure, a tape drive is expensive, but the tapes are quite
> cheap.
>
> The cheapness of the tapes allows you to use several for different
> backup strategies (weekly and daily incremental and monthly full backup, for
> example) and to keep older data on saved tapes.
Tape isn't as much cheaper as it used to be. IDE disks are going for
$132 for 120GB. You can do RAID 5 with 8 drives for around $1.25 a
gig. You can get tapes (DDS3) for about $.50 a gig (don't pay attention
to the max compressed size) but that doesn't factor in a lot of other
costs. Drives aren't cheap and neither are the people who'll be
changing your drives. You also won't pack that tape as well as you will
disks. When you cycle out your weekly set you'll be throwing away
unused capacity. Disks stay around until you use every last bit.
> It's easier. There are programs like Arkeia (free for one linux server and
> two clients (win32/linux)), that makes tape and backup management a few
> clicks (but a read of documentation is still needed). As a side note,
> Arkeia supports direct dumping and backup from serveral rdbms, postgresql
> included.
Not really. Just use pg_dump to dump to disk and write yourself a
wrapper to figure out a filename based on the date.
> It's more reliable. If the backup disk fails, all backup is lost and its
> substitution an hassle. If a tape breaks you still have the other tapes
> for last week/day/etc.. Still, a tape drive can fail or a tape can screw
> the drive, but I haven't heard of anyone to whom that has happened. In
> hardware world, quality normally comes with an higher price.
You won't lose data from disk if you use RAID.
I know it's hard to believe, but we're actually saving money dumping to
disk. We needed a new system and the decision of tape vs disk was made
based on cost alone.
Another benefit of disk is that all your backups are online. That comes
in really handy for postgres because you can do incs very easily with
diff. If you're looking to save money on postgres backups a good place
to start is not saving an epoch each day.
--Nate
--Nate
On 17 Jul 2002, Nathan Mueller wrote: > I know it's hard to believe, but we're actually saving money dumping to > disk. We needed a new system and the decision of tape vs disk was made > based on cost alone. It's easy to believe you're saving money; the question is, how far are your backup disks from the machine they're backing up? Some people have a requirement for off-site backups so that they they still have their data if their building burns down. However, you're right that disk has many advantages. A big one is speed. You can do the backups themselves much faster and even skip the verify pass (if you're using tape, you really should be verifying you can read every single one you write), and you can restore a lot faster, too. When I've got the money, I keep a week's worth of backup on disk at the site, and use tapes only for retreiving stuff older than that and for off-site backups. cjs -- Curt Sampson <cjs@cynic.net> +81 90 7737 2974 http://www.netbsd.org Don't you know, in this new Dark Age, we're all light. --XTC
I used Exabyte tapes for severeal years, very reliable. Sometimes there were problems with media from different vendors. I usually used 2 tapes for parallel backup/recover sessions and for fault tolerance. I would backup to harddisks only in rare cases and I don't user CDs etc. for backups. Egon Tony Reina wrote: > Yes, I'm a little wary of hard disk based systems as my sole backup. I > prefer something that allows me to have at least 2-3 different media > backups (e.g. one this week, one last week, and one the week before last). > Also, I like to be able to take a copy of the media home just because I'm > paranoid that the lab will burn down or something silly like that ;>) > > I've been looking at the Exabyte systems which hold 80G/160G and write at > about 10G/hour. Anyone have experience with these? How onerous is it to > look at the table of contents or restore a specific file from these tapes? > > -Tony > > At 11:35 PM 7/17/02 +0100, strange@nsk.yi.org wrote: > >On Wed, Jul 17, 2002 at 04:48:10PM -0500, Nathan Mueller wrote: > > > > Are there > > > > systems like with these features that could handle say 20 Gigs per > > > > media? > > > > > > Where I work we just started using a new disk based backup system that > > > we wrote in-house. Disk is a little bit cheaper then tape -- plus much > > > faster. Another bonus is that your compression rate does not depend on > > > how fast your data is flowing. I'd suggest you buy a few 160GB IDE > > > drives and just dump your data there. It's faster, easier and you don't > > > need to change tapes. If you're interested in the source to our system > > > just let me know. > > > >I advise against hard disk based backups: > > > >It's not cheap, sure, a tape drive is expensive, but the tapes are quite > >cheap. > > > >The cheapness of the tapes allows you to use several for different > >backup strategies (weekly and daily incremental and monthly full backup, for > >example) and to keep older data on saved tapes. > > > >It's easier. There are programs like Arkeia (free for one linux server and > >two clients (win32/linux)), that makes tape and backup management a few > >clicks (but a read of documentation is still needed). As a side note, > >Arkeia supports direct dumping and backup from serveral rdbms, postgresql > >included. > > > >It's more reliable. If the backup disk fails, all backup is lost and its > >substitution an hassle. If a tape breaks you still have the other tapes > >for last week/day/etc.. Still, a tape drive can fail or a tape can screw > >the drive, but I haven't heard of anyone to whom that has happened. In > >hardware world, quality normally comes with an higher price. > > > >That being said, I don't have a tape. Too much for my pocket. :) > >So I made a script that creates, compress, splits and burns backups to > >cds. I'll made a switch to dvds when the price for dvd recorders drops to > >EUR 100 and dvd blank discs to EUR 2... > > > >Your solution is still a valid one, but more a special case for those > >without too much $$... > > > >Regards, > >Luciano Rocha > > > >-- > >Consciousness: that annoying time between naps. > > ---------------------------(end of broadcast)--------------------------- > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
Hi -*- Nathan Mueller <nate@cs.wisc.edu> [ 2002-07-18 00:18 ]: > You won't lose data from disk if you use RAID. Well... rm, anyone ? :) RAID protects you from drive failures, sure, but not many other factors such as human mistake, software errors, compromisedbox etc. I personally do a local pg_dump daily, and the entire machines are then backup up by Tivoli Storage Manager. Regards, Tolli -- Thorhallur Halfdanarson tolli@margmidlun.is Systems Administrator / Network Engineer - MInet Margmidlun hf. www.mi.is Tel: (+354) 575-7078 Fax: (+354) 575-7001
You might try looking into DLT technology. DLT tape drives will give you 80Gb storage with ~6-12MB/s throughput. They are an industry standard supported by the heavyweights (HP/IBM/Dell) but a little more pricey than your current DDS technology. If you are looking for speed AND storage, I would suggest a combination of disk and tape. Back you 'production' data to a separate 'backup' drive then write the 'backup' drive contents to tape. If you have money to burn and require very little downtime during backup, place your data on a mirrored disk. During backup time split the mirror and backup the stale mirror to tape. Once you are done, re-sync the stale mirror to your production mirror (which can be done online). My CDN$.02, K. Tony Reina wrote: > I'm looking into new ways of backing up the data in our lab, including > the PostgreSQL database. Currently, we have a single DDS-2 tape drive > capable of holding 8Gig compressed. However, it is slow (i.e. takes a > day to backup about 40 Gig of information), spans multiple tapes, and > makes it hard to find just a single file or two on the backup (i.e. I > have to go through many tapes before I can find and extract one of two > files). Our CDRW backups are easier to manage and relatively fast, > but require dozens of CDRWs. > > Can anyone make suggestions on backup systems? I was thinking that > some sort of DVD writing system would be good for accessing one or two > files in the backup quickly. It would probably also complete backups > faster. However, I think DVD's only hold a few Gigs on them. Are there > systems like with these features that could handle say 20 Gigs per > media? > > Thanks. > -Tony > > ---------------------------(end of broadcast)--------------------------- > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org > > -- Kurt Gunderson Senior Programmer Applications Development Lottery Group Canadian Bank Note Company, Limited Email: kgunders@cbnlottery.com Phone: 613.225.6566 x326 Fax: 613.225.6651 http://www.cbnco.com/ "Entropy isn't what is used to be" Obtaining any information from this message for the purpose of sending unsolicited commercial Email is strictly prohibited. Receiving this email does not constitute a request of or consent to send unsolicited commercial Email.
On Thu, 18 Jul 2002, Kurt Gunderson wrote: > You might try looking into DLT technology. DLT tape drives will give > you 80Gb storage with ~6-12MB/s throughput. They are an industry > standard supported by the heavyweights (HP/IBM/Dell) but a little more > pricey than your current DDS technology. > > If you are looking for speed AND storage, I would suggest a combination > of disk and tape. Back you 'production' data to a separate 'backup' > drive then write the 'backup' drive contents to tape. > > If you have money to burn and require very little downtime during > backup, place your data on a mirrored disk. During backup time split > the mirror and backup the stale mirror to tape. Once you are done, > re-sync the stale mirror to your production mirror (which can be done > online). > > Tony Reina wrote: > > > I'm looking into new ways of backing up the data in our lab, including > > the PostgreSQL database. Currently, we have a single DDS-2 tape drive > > capable of holding 8Gig compressed. However, it is slow (i.e. takes a > > day to backup about 40 Gig of information), spans multiple tapes, and > > makes it hard to find just a single file or two on the backup (i.e. I > > have to go through many tapes before I can find and extract one of two > > files). Our CDRW backups are easier to manage and relatively fast, > > but require dozens of CDRWs. > > > > Can anyone make suggestions on backup systems? I was thinking that > > some sort of DVD writing system would be good for accessing one or two > > files in the backup quickly. It would probably also complete backups > > faster. However, I think DVD's only hold a few Gigs on them. Are there > > systems like with these features that could handle say 20 Gigs per > > media? Weigh your needs vs. technology available first. If you need speed and large capacity and want to use classic backup media (ie: sequential), look no further than some of the high end tape drives. DLT is at the lower end of "high end", but it will still work OK. There's also LTO, STK, IBM 3590... You can spend as much as you want to spend. If speed is your goal beyond anything else, get IBM 3590 (others may be as fast, but I know this drive is pretty advanced as far as features). If you want price/performance, get DLT. I don't really see any of the consumer available optical devices (ie DVD-RW or CD anything) as viable backup devices unless your data will fit on them and speed is no concern. They are slow and rather small compared to almost any modern tape drive. DDS may be excepted, I think those are pretty slow but I haven't looked into them lately. Based on what the last post said, you may want to go with a backup program like Amanda which will make tapes for you after first staging the data in a disk holding area. This works at the filesystem level though, so you'd have to have your database data consolidated carefully. Andy -- acruhl@sdf.lonestar.org SDF Public Access UNIX System - http://sdf.lonestar.org
Thanks everyone. I think I'll go with one of the tape solutions (DLT, Exabyte VXA-2, or AIT) after all. If I can get the 80G native/160 Gig compressed, then I can probably get buy with one or two tapes. That should make my single-file restores a little less onerous. -Tony kgunders@cbnlottery.com (Kurt Gunderson) wrote in message news:<3D36CE72.6070308@cbnlottery.com>... > You might try looking into DLT technology. DLT tape drives will give > you 80Gb storage with ~6-12MB/s throughput. They are an industry > standard supported by the heavyweights (HP/IBM/Dell) but a little more > pricey than your current DDS technology. > > If you are looking for speed AND storage, I would suggest a combination > of disk and tape. Back you 'production' data to a separate 'backup' > drive then write the 'backup' drive contents to tape. > > If you have money to burn and require very little downtime during > backup, place your data on a mirrored disk. During backup time split > the mirror and backup the stale mirror to tape. Once you are done, > re-sync the stale mirror to your production mirror (which can be done > online). > > My CDN$.02, > K. >
At 07:43 PM 7/17/02 , Curt Sampson wrote:
>if you're using tape, you really should be verifying you can read every
>single one you write
Professional quality tape drives do read-after-write with a separate head
during the same pass, so the tape has already been read.
If you're using quarter-inch-cartridges or something like it, then yes, by
all means to a test read. But DDS and DLTs can detect a write error and
correct it on the fly.
-crl
--
Chad R. Larson (CRL22) chad@eldocomp.com
Eldorado Computing, Inc. 602-604-3100
5353 North 16th Street, Suite 400
Phoenix, Arizona 85016-3228
Hi, > It's easy to believe you're saving money; the question is, how far are > your backup disks from the machine they're backing up? Some people have > a requirement for off-site backups so that they they still have their > data if their building burns down. Yep. In our situation tapes have to be stored inside a safe. At least one recent tape must be off-site, etc. Not only for when the building burns down, but also because someone might break in and steal equipment. > However, you're right that disk has many advantages. A big one is speed. > You can do the backups themselves much faster and even skip the verify > pass (if you're using tape, you really should be verifying you can read > every single one you write) I realy like tapedrives with built-in read-after-write. Our AIT2 drive automatically checks if everything is written reliably, and if not, it automatically rewrites the data. Makes me sleep better :) Sander.
<<TESTIMONIAL ON>>
I used DLT in industry and didn't like it due to excessive rewinding
and a pain to deal with.
I use a HP DDS drive purchased a year or so ago on one older Sun box,
but forgot which DDS version. No problems ever, but I don't buy
noname tapes. Just wait for a sale and buy a bunch...
Won a free external VXA Ecrix drive at a CLUG (Linux) meeting.
Tapes cost much more than DDS ones, but they have extra
redundancy in their methods, so theoretically, years from
now, you will have better luck reading all the tape. And you
can supposedly boil the tape and still read it... :-)
But, I'm constantly re-creating the world, and not interested in
archiving every database change for historical reasons, so I don't
chew through tapes. I just want to be able to re-create the world
even if there is some catastrophic failure or worst-case
hack scenario. With tape and dedicated disks for backup,
I don't worry.
Boiling tapes notwithstanding. There was a two-tape VXA version
bundled with a bunch of tapes for a decent price awhile back,
via a promotion, before their new technology came out, before
the Exabyte merger took hold in the product lines. Have one of
those, so you can write two tapes serially or in parallel. Have
seen those cartridge tape changers fail in industry. There we always
purchased tape drives in pairs, even external ones, even if one sat
on a shelf. [Unless it was some big robotic multi-tape system that
cost a fortune...] The VXA stuff works fine with Linux. But I don't
entirely trust any tape for backups, so I have a few extra SCSI disk
drives just for overlapping backups. The kind that pop out of the
enclosure, so they can be stored elsewhere, like tapes.
I think the proprietary vs. non-proprietary thing with tape is not all a
proprietary/non-proprietary issue. Why do I say this? (i) Because in
reality tape drive heads can drift over time, so you might have drifted
away from a non-proprietary setup, and then the effect is the same.
Maybe that's just old school experiences, and engineering now is
good enough so that doesn't happen. And (ii) they're all manufactured
by some company, not like software, and each by definition has
quality control issues at some level or another. However, cost of tape
should be a concern, if you really go through them in the backup scheme.
Dust and particulate matter around a tape drive is something to
look out for!!! Those particles may hose your database backup someday.
And keep them away from monitors and cheap power supplies.
Haven't missed the money of industry, IT departments, Oracle, Sybase, Ingres,
CA-Ingres (reamola) and all that, and this Linux stuff is great for saving the
contents of my change-purse. A necessity at universities. No free beer on
Fridays, but much more sanity in my vicinity with PostgreSQL.
No per-seat, per-CPU, per-anything headaches. I've been watching it
develop since Postgres95. Had the air of dignity about it back then too,
but back then I was just a voyeur.
douglas
p.s. I've *never* been let down by tar.
p.p.s. GNU tar is even better than the old tar.
Tony Reina wrote:
> Thanks everyone. I think I'll go with one of the tape solutions (DLT,
> Exabyte VXA-2, or AIT) after all. If I can get the 80G native/160 Gig
> compressed, then I can probably get buy with one or two tapes. That
> should make my single-file restores a little less onerous.
>
> -Tony
At 01:54 PM 7/18/02 , Douglas Trainor wrote:
>Because in reality tape drive heads can drift over time, so you might have
>drifted away from a non-proprietary setup, and then the effect is the same.
DLTs have a special "calibration" section at the head of the tape. Upon a
mount for read, the drive will re-read that section and scootch the head up
or down until it gets a good read. Another of the reasons the DLT
consortium claims a 30 year archival life.
-crl
--
Chad R. Larson (CRL22) chad@eldocomp.com
Eldorado Computing, Inc. 602-604-3100
5353 North 16th Street, Suite 400
Phoenix, Arizona 85016-3228