Re: Bidirectional replication

On Fri, May 06, 2011 at 09:15:37AM -0400, Vick Khera wrote:
>
> Oh... I remember erserver.  It served us well for about 2 years for a
> simple, not very high-velocity database that was 99.44% read-only.  I
> did have to monitor it closely and restart regularly.  At least with
> slony I don't worry about it crashing out from under me... :)

<oldtimer style="doddering">

To be fair to it, erserver managed to live through the launch of the
.info registry and its expansion, as well as the technical takeover of
.org by Afilias.  These were both fairly high-volume systems.  It was
plenty challenging to use for this arrangement, but it did manage to
do the job.

The big flaw in it was the decision on the part of the "architect" at
the time to use Java.  Java was picked because, well, because it said
Enterprise in the name, as far as I could tell.  This was made
considerably worse by the decision to employ a programmer (who shall
remain nameless) on the project who didn't know anything about
Postgres.  (He was gone from the project and from anything to do with
the Afilias systems by the end of 2001, IIRC.)

This was all to re-implement POC code that Vadim Mikheev had written,
which was an extenion of the rserv code.  That used named pipes, and
the guy who got to make these decisions thought named pipes were for
weenies.  All the deep and clever tricks in erserver were ones Vadim
wrote.  (I suspect if you ask Jan, he'll tell you that some of the
Slony code, especially in the way the triggers work, was also inspired
by that code.)

The fellow who reimplemented all this in Java desperately wanted to
use multiple threads to get all the data, so he could run in parallel.
(He had a mania for threads.  He had an error reporting thread that
was supposed to handle all exceptions.  He stubbed it out and then
never wrote the exception handler, which meant that every single
exception you ever got from the code came from the same line number
and said "Exception" but nothing more.  This was extremely fun for
debugging.  To this day, whenever someone starts telling me that
threads are the obvious solution to some problem, I want to quiz them
really hard.  I am pretty sure I can count in single digits the number
of people I've ever met who actually know what they're doing with
threads.)

His plan, in fact, was to open one connection to the database for
every table in the replication set.  When I told him that this was a
non-starter (we'd have used 50 or 60 connections just for this -- in
the days when the JDBC driver would issue BEGIN and sit there like a
dummy, blocking all your vacuums), he didn't revise his plan, so we
ended up with a system that could in principle pull inconsistent views
of the data into the replication engine.  The way to fix this, as it
turned out, was to force the connection pool only to use one
connection.  Anyway, because of this multithreading plan, he couldn't
sort the data to be applied in a reasonable way in the database, so
he'd pull all the pending data into the replication engine, then sort
it, and then apply it on the slave.  Well, no, actually.  He'd sort
and copy it, because if he had more than one slave the sort might have
to be different.  Or something -- I never actually understood the
reason for this, and I stopped trying after all my teeth had to be
replaced from the grinding.  But it meant that if you got far enough
behind, the replication engine would blow up because it ran out of
memory it could allocate.  IIRC, the VM was limited to 4G total.

We had a large number of mitigation strategies we used to catch the
possibility that the replication engine would get behind.  I'm pretty
sure it was the analysis of on-call billing hours that ultimately
convinced management we needed to start over.  I know that some people
(I am among them) find Slony complicated and hard to manage.  I can
tell you, however, that compared to the stone tools we had for
replicating in 2001, it was a dream.

</oldtimer>

A

--
Andrew Sullivan
ajs@crankycanuck.ca