Re: Proposal: Adding json logging

John W Higgins wrote:
> On Sun, Apr 15, 2018 at 11:08 AM, David Arnold <dar@xoe.solutions> wrote:
> 
> > >This would appear to solve multiline issues within Fluent.....
> > >https://docs.fluentd.org/v0.12/articles/parser_multiline
> >
> > I definitely looked at that, but what guarantees do I have that the
> > sequence is always ERROR/STATEMENT/DETAIL? And not the other way round?
> 
> Have you asked that question? You seem to at least have opened the source
> code - did you try to figure out what the logging format is?

I looked at this a couple of days ago.  I think parsing with this
library is possible to a certain extent, and the problems stem from
limitations of the library.  So, it turns out that the firstline can be
set to a single regex that searches for PANIC, FATAL, ERROR, WARNING,
LOG, NOTICE, DEBUG.  That's always the first line in any postgres log
event.

A log event contains some subsequent lines.  Those start either with a
tab (which is a continuation of the previous line) or with one of
DETAIL, CONTEXT, HINT, STATEMENT, QUERY.  This seems very simple to
parse (just add lineN patterns for those), *except* that the messages
can be multiline too; and where would you assign the continuation lines
for each of those?  parser_multiline does not support that.

Another thing worth keeping in mind is that you need to change the regex
depending on log_line_prefix, which sounds very painful.

All in all, the best approach might be to create a specific
parser_postgresql.rb plugin.  Seems much easier to write two dozen lines
of Ruby than change all of PostgreSQL's logging infrastructure.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services