Обсуждение: DBmirror replication - replacement for DBMirror.pl

Not sure whether this is any use to anyone, or whether this is the right
list to post to but...

I've just released a C++ implementation of the DBMirror.pl script as
part of Whitebeam (http://www.whitebeam.org). We had *real* performance
issues with the Perl implementation replicating large fields. It's an
almost drop in replacement - except configuration file format is
difffernt. It's also more fault tolerant than the Perl implementation
and has some parameters to help distribute the replication load over
time (if you delete 10,000 rows from a table, you don't really want
you're main application to grind to a halt as replication soak up most
of your resource!)

I needed to do this quickly - so it links to the utility classes in
Whitebeam, things like a string class etc. It wouldn't be too difficult
to decouple these and if there is any interest I'll do that when I get a
little spare time. Once it's built though it's entirely generic and
doesn't use anything else from Whitebeam.

If anyone has contact information for the original DBMirror author then
I'd like to get in touch.

Would like to get feedback from anyone using DBmirror or thinks this
would be useful..

Background
========

Our Whitebeam application server uses PostgreSQL for back-end data
storage. We originally used IBM DB2, but ported to Postgres about 3
years ago, but we never sorted out replication (partly because we were
using Large Objects). I recently sorted that out and looked around for a
replication scheme to use - first choice being Slony-I. I couldn't get
it to work - the configuration scheme just kept claiming half our schema
tables didn't have primary keys (they did!). Also the documentation was
virtually non-existent.

Dropping back to DBmirror (which comes with the PostgreSQL distribution
in the 'contrib' directory) worked fine-ish. Unfortunately the
replication Perl script was un-usably slow, taking minutes to replicate
a 100K BYTEA field, which we used to store images. The replication Perl
script seemed to be rather inefficient, using a lot of regular
expressions to decode field values etc. Perl isn't something I felt too
confident in - and I needed a solution quickly and hence the C++
implementation

Pete
--
Peter Wilson
YellowHawk : http://www.yellowhawk.co.uk
Whitebeam : http:/www.whitebeam.org
-----

Just to add - the replacement for DBMirror.pl automatically works out the schema in use and will replicate using DBmirror from 7.x or from 8.x (table names all changed!). This is done transparently.

I also wrote a short article at http://www.whitebeam.org/library/guide/TechNotes/replicate.rhtm

Pete

Peter Wilson wrote:

Not sure whether this is any use to anyone, or whether this is the right list to post to but...

I've just released a C++ implementation of the DBMirror.pl script as part of Whitebeam (http://www.whitebeam.org). We had *real* performance issues with the Perl implementation replicating large fields. It's an almost drop in replacement - except configuration file format is difffernt. It's also more fault tolerant than the Perl implementation and has some parameters to help distribute the replication load over time (if you delete 10,000 rows from a table, you don't really want you're main application to grind to a halt as replication soak up most of your resource!)

I needed to do this quickly - so it links to the utility classes in Whitebeam, things like a string class etc. It wouldn't be too difficult to decouple these and if there is any interest I'll do that when I get a little spare time. Once it's built though it's entirely generic and doesn't use anything else from Whitebeam.

If anyone has contact information for the original DBMirror author then I'd like to get in touch.

Would like to get feedback from anyone using DBmirror or thinks this would be useful..

Background
========

Our Whitebeam application server uses PostgreSQL for back-end data storage. We originally used IBM DB2, but ported to Postgres about 3 years ago, but we never sorted out replication (partly because we were using Large Objects). I recently sorted that out and looked around for a replication scheme to use - first choice being Slony-I. I couldn't get it to work - the configuration scheme just kept claiming half our schema tables didn't have primary keys (they did!). Also the documentation was virtually non-existent.

Dropping back to DBmirror (which comes with the PostgreSQL distribution in the 'contrib' directory) worked fine-ish. Unfortunately the replication Perl script was un-usably slow, taking minutes to replicate a 100K BYTEA field, which we used to store images. The replication Perl script seemed to be rather inefficient, using a lot of regular expressions to decode field values etc. Perl isn't something I felt too confident in - and I needed a solution quickly and hence the C++ implementation

Pete
--
Peter Wilson
YellowHawk : http://www.yellowhawk.co.uk
Whitebeam : http:/www.whitebeam.org
-----


---------------------------(end of broadcast)---------------------------
TIP 8: explain analyze is your friend