Re: BDR Selective Replication

On 4/27/15 7:54 PM, Craig Ringer wrote:
>     If 'default replication set' is the idea of "here's what tables
>     *should* be getting replicated regardless of whether that's
>     happening or not", it'd be great if that was done so it could be
>     split out on it's own at some point. It's a problem that affects all
>     replication systems.
>
>
> It wasn't, but that's an interesting idea.
>
> You need  away to identify peer nodes in an abstract way before you can
> really define sets of which nodes should get which tables. So I think
> replication identifiers ( https://commitfest.postgresql.org/4/161/ ) are
> a pre-requisite for that though, and one that's proving difficult to get
> in.

Perhaps... different replication systems probably use different methods
to identify, so presumably there'd need to be some way to map a generic
identifier into an appropriate identifier for whatever replication
system you're using.

> I think any sort of replication sets is likely to have similar problems,
> especially the "no in-core user" problem. There's nothing fundamentally
> impossible about filtering WAL sent to physical downstreams over
> streaming replication to include only replicated tables and the
> catalogs, though, so perhaps there could be an in-core user for it.

Oh, I wasn't thinking this needed to be in-core. I think it'd be a lot
easier to develop it as an extension to start with... certainly a lot
less headache ;) If it becomes popular then it'll be a lot easier to get
it added.

> In BDR we're currently (ab)using security labels to tag tables with
> their replication sets, but I'd love to have a proper way to do that. As
> I recall the prior approach, of allowing custom relation options, was
> rejected on -hackers.
>
> How would you want to go about storing and tracking the information? A
> new catalog? The other issue for in-core replication sets would probably
> be making it foreign-key aware, so replication of a table transitively
> requires replication of its references.

As you said, we'd need a way to identify replication nodes. We might
also need/want a way to specify topology. I don't think topology would
be too hard (presumably it's either a single 'parent' node, or a list of
peers). What might be more interesting is dealing with different systems
methods of identifying nodes.

You'd want a way to define different sets and associate them with nodes.
A node could be a provider, subscriber, or both. I think some
replication systems support 'pass through' as well, where the node
passes data downstream but doesn't apply it itself. Or it could be
multi-master and possibly a provider to read-only subscribers.

Finally you'd need to associate tables and sequences with a set. I agree
you'd want to look at FKs. I'd also like to be able to define rules for
a set, like "include everything in this schema, unless the first
character is _".
--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com