Re: Changeset Extraction Interfaces

On 2013-12-14 11:50:00 -0500, Robert Haas wrote:
> On Fri, Dec 13, 2013 at 9:14 AM, Andres Freund <andres@2ndquadrant.com> wrote:
> > But can you imagine those users needing an exported snapshot? I can
> > think of several short-lived usages, but all of those are unlikely to
> > need a consistent view of the overall database. And those are less
> > likely to be full blown replication solutions.
> > I.e. it's not the DBA making that decision but the developer making the
> > decision based on whether he requires the snapshot or not.
> 
> Well, it still seems to me that the right way to think about this is
> that the change stream begins at a certain point, and then once you
> cross a certain threshold (all transactions in progress at that time
> have ended) any subsequent snapshot is a possible point from which to
> roll forward.

Unfortunately it's not possible to build exportable snapshots at any
time - it requires keeping far more state around since we need to care
about all transactions, not just transactions touching the
catalog. Currently you can only export the snapshot in the one point we
become consistent, after that we stop maintaining that state.

I see pretty much no chance to change that without major effort that's
imo clearly out of scope for 9.4. Maybe we want to provide that at some
point, but it's going to be a bit.
Also note that in order to import a snapshot, the exporting snapshot
still has to be alive, in the same transaction that has exported the
snapshot. So, until the snapshot has been imported, replication cannot
progress. That's existing limitations from the snapshot import code, but
the reasoning for them make it unlikely that we can easily change them

Under that light, usecases like your example won't be able to benefit
from getting changes before the snapshot, or do you still see usecases
for that?

I've already prototyped the "quickstart" option that doesn't allow
exporting a snapshot, although I don't know how the UI for it is going
to look.

> > I think I am just scared of needing to add more features before getting
> > the basics done and in consequence overrunning 9.4...
> 
> I am sensitive to that.  On the other hand, this API is going to be a
> lot harder to change once it's released, so we really need to avoid
> painting ourselves into a corner with v1.

I think we need to accept the fact that quite possibly we won't get
everything right in 9.4. Obviously we should strive to avoid that, but
trying to design things perfectly usually ends in not getting there.
Yes, the API changes for FDWs caused pain. But there's no way we would
have the current features if we'd tried to introduce the correct API
allowing them from the get go.

>  As far as high-level design
> concerns go, there are three things that I'm not happy with:
> 
> 1. Slots.  We know we need physical slots as well as logical slots,
> but the patch as currently constituted only offers logical slots.

Well, then tell me the way you want to go forward on that end. I can
make the slot interface more generic if we know exactly what we need,
but I doesn't seem fair to take this patch hostage until I develop a
separate not so small feature. Why is that my task?
Because I think it's important, and because by now I know the related
code pretty well by now, I am willing to provide the parts of the that
prevent required WAL from being deleted, peg xmin and report the current
state to SQL, but somebody else is going to have to the rest.

> s/pg_decoding_slot/pg_logical_stream/?

Works for me.

Regards,

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services