Re: preserving forensic information when we freeze

On 2014-01-02 12:46:34 -0500, Tom Lane wrote:
> Andres Freund <andres@2ndquadrant.com> writes:
> > On 2014-01-02 09:40:54 -0500, Tom Lane wrote:
> >> Actually, I thought the function approach was a good proposal.  You are
> >> right that func(tab.*) isn't going to work, because it's going to get a
> >> Datum-ified tuple not a pointer to raw on-disk storage.  But an inspection
> >> function that's handed a ctid could work.
> 
> > Well, we discussed that upthread, and the overhead of going through a
> > function is quite noticeable because the tuple needs to be fetched from
> > the heap again.
> 
> Yeah, I read those results, but that seems like it could probably be
> optimized.  I'm guessing the function was doing a new heap_open every
> time?  There's probably a way around that.

How? Everything I can think of is either fragile or too ugly.

> In any case, upon further reflection I'm not convinced that doing this
> with a SELECT-based query is the right thing, no matter whether the query
> looks at a function or a system column; because by definition, you'll only
> be able to see tuples that are visible to your current snapshot.

I think it really depends - quite often you really want to just
investigate tuples you actually can see, because you can easily write
normal queries and such. It sure wouldn't replace pageinspect.

>  For real
> forensics work, you need to be able to see all tuples, which makes me
> think that something akin to pgstattuple is the right API; that is "return
> a set of the header info for all tuples on such-and-such pages of this
> relation".  That should dodge any performance problem, because the
> heap_open overhead could be amortized across lots of tuples, and it also
> sidesteps all problems with adding new system columns.

The biggest problem with such an API is that it's painful to use - I've
used pageinspect a fair bit, and not being able to easily get the
content of the rows you're looking at makes it really far less useful in
many scenarios. That could partially be improved by a neater API
(returning t_self would help greatly, accepting a page without a wrapper
function as well), but it still pretty fundamentally sucks.

And I really don't see any page-at-a-time access that's going to be
convenient. A big step would be returning the tuple's contents in a
normal fashion, but that's not easy without big notational overhead.

> > Upthread there's a POC patch of mine, that started to explore what's
> > necessary to simply never store system columns (except maybe oid) in
> > pg_attribute. While it passes the regression tests it's not complete,
> > but the amount of work looks reasonable.
> 
> I think this will inevitably break a lot of code, not all of it ours,
> so I'm not in favor of pursuing that direction.

Are you thinking of client or extension code? From what I've looked at I
don't think it's all that likely too break much of either. Extension
code isn't likely to do its own catalog lookups and thus doesn't really
care. Most client code isn't interested in system catalog attribute
numbers - the few examples of client code looking at pg_attribute I
remember all explicitly excluded attnum < 0.

It would make pg_attribute a fair bit smaller, especially on systems
with lots of narrow relations. There's also the possibility of only
going that route for new system catalog attributes, and leave the old
ones where they are and removed them in a release or two.

Greetings,

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services