Re: stats for network traffic WIP

On Wed, Dec 18, 2013 at 8:47 AM, Stephen Frost <sfrost@snowman.net> wrote:
> Agreed.  My other thought on this is that there's a lot to be said for
> having everything you need available through one tool- kinda like how
> Emacs users rarely go outside of it.. :)  And then there's also the
> consideration that DBAs may not have access to the host system at all,
> or not to the level needed to do similar analysis there.

I completely agree with this, and yet I still think we should reject
the patch, because I think the overhead is going to be intolerable.

Now, the fact is, the monitoring facilities we have in PostgreSQL
today are not nearly good enough.  Other products do better.  I cringe
every time I tell someone to attach strace to a long-running autovac
process to find out what block number it's currently on, so we can
estimate when it will finish; or every time we need data about lwlock
contention and the only way to get it is to use perf, or recompile
with LWLOCK_STATS defined.  These are not fun conversations to have
with customers who are in production.

On the other hand, there's not much value in adding monitoring
features that are going to materially harm performance, and a lot of
the monitoring features that get proposed die on the vine for exactly
that reason.  I think the root of the problem is that our stats
infrastructure is a streaming pile of crap.  A number of people have
worked diligently to improve it and that work has not been fruitless,
but the current situation is still not very good.  In many ways, this
situation reminds me of the situation with EXPLAIN a few years ago.
People kept proposing useful extensions to EXPLAIN which we did not
adopt because they required creating (and perhaps reserving) far too
many keywords.  Now that we have the extensible options syntax,
EXPLAIN has options for COSTS, BUFFERS, TIMING, and FORMAT, all of
which have proven to be worth their weight in code, at least IMHO.

I am really not sure what a better infrastructure for stats collection
should look like, but I know that until we get one, a lot of
monitoring patches that would be really nice to have are going to get
shot down because of concerns about performance, and specifically
stats file bloat.  Fixing that problem figures to be unglamorous, but
I'll buy whoever does it a beer (or another beverage of your choice).

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company