Re: Auditing extension for PostgreSQL (Take 2)

On Thu, May  7, 2015 at 03:41:13PM -0400, Stephen Frost wrote:
> Bruce,
> > What is our history of doing things in contrib because we are not sure
> > what we want, then moving it into core?  My general recollection is that
> > there is usually something in the contrib version we don't want to add
> > to core and people are locked into the contrib API, so we are left
> > supporting it, e.g. xml2, though you could argue that auditing doesn't
> > have application lock-in and xml2 was tied to an external library
> > feature.
> 
> That's exactly the argument that I'd make there.  My recollection is
> that we did move pieces of hstore and have moved pieces of other contrib
> modules into core; perhaps we've not yet had a case where we've
> completely pulled one in, but given the relatively low level of
> dependency associated with pgAudit, I'm certainly hopeful that we'll be
> able to here.  Lack of history which could be pointed to that's exactly
> what I'm suggesting here doesn't seem like a reason to not move forward
> here though; the concept of having a capability initially in contrib and
> then bringing it into core has certainly been discussed a number of
> times on other threads and generally makes sense, at least to me,
> especially when there's little API associated with the extension.

OK, I am just asking so we remember this can go badly, not that it will.

> > I guess the over-arching question is whether we have to put this into
> > contrib so we can get feedback and change the API, or whether using from
> > PGXN or incrementally adding it to core is the right approach.
> 
> I'm surprised to hear this question of if we "have to" do X, Y, or Z.
> pgAudit brings a fantastic capability to PostgreSQL which users have
> been asking to have for many years and is a feature we should be itching
> to have included.  That we can then take it and incrementally add it to
> core, to leverage things which are only available in core (as discussed
> last summer, including grammar and relation metadata), looks to me like
> a great direction to go in and one which we could use over and over to
> bring new features and capabilities to PG.
> 
> Lack of auditing is one of the capabilities that users coming from other
> large RDBMS's see as preventing their ability to migrate to PostgreSQL.
> Other databases (open and closed source) have it and have had it for
> years and it's a serious shortcoming of ours that makes users either
> stick with their existing vendor or look to other closed-source or even
> open-source solutions.

Yes, more extensive auditing is definitely needed.

> > > involved in this discussion will be also.  Additionally, as discussed
> > > last summer, we can provide a migration path (which does not need to be
> > > automated or even feature compatible) from pgAudit to an in-core
> > > solution and then sunset pgAudit.
> > 
> > Uh, that usually ends badly too.
> 
> I'm confused by this, as it was the result of our discussion and your
> suggestion from last summer: 20140730192136.GM2791@momjian.us
> 
> I certainly hope that hasn't substantially changed as that entire
> discussion is why we're even able to have this discussion about
> including pgAudit now.  I was very much on-board with trying to work on
> an in-core solution until that thread convinced me that the upgrade
> concerns which I was worried about wouldn't be an issue for inclusion of
> an extension to provide the capability.

I had forgotten about that.  Yes, pg_upgrade can easily do this.

> > > Building an in-core solution, in my estimation at least, is going to
> > > require at least a couple of release cycles and having the feedback from
> > > users of pgAudit will be very valuable to building a good solution, but
> > > I don't believe we'll get that feedback without including it.
> > 
> > See above --- is it jump through the user hoops and only then they will
> > use it and give us feedback?  How motivated can they be if they can't
> > use the PGXN version?
> 
> Why wouldn't we want to include this capability in PG?  I also addressed
> the "why not PGXN" above.  It it not a lack of motivation but the entire
> intent and design of the PGXN system which precludes most large
> organizations from using it, particularly for sensitive requirements
> such as auditing.

So they trust the Postgres, but not the PGXN authors --- I guess legally
that makes sense.

> > The bottom line is that for the _years_ we ship pg_audit in /contrib, we
> > will have some logging stuff in postgresql.conf and some in
> > contrib/pg_audit and that distinction is going to look quite odd.  To
> > the extent you incrementally add to core, you will have duplicate
> > functionality in both places.
> 
> That's entirely correct, of course, but I'm not seeing it as an issue.
> I'm certainly prepared to support shipping pgAudit in contrib, as are
> others based on how this feature has been developed, for the years that
> we'll have 9.5, 9.6 (or 10.0, etc) supported- and that's also another
> reason why users will use it when they wouldn't use something on PGXN.
> 
> Further, I look forward to working incrementally to bring similar
> capability into core, but I suspect those increments will largely be in
> the infrastructure until we reach the point where we're able to provide
> the user-facing bits, which is quite likely to go in all at once and
> allow us a clear upgrade path from one to the other.  Perhaps that's
> optimistic, but we do tend to try and bring things in as whole
> capabilities rather than bits and pieces and I don't expect us to need
> to do it differently here.

OK, I just felt I had to ask those questions so we know where the
pitfalls are --- over-optimism always sets of alarms for me.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + Everyone has their own god. +