John W Higgins wrote:
> On Sun, Apr 15, 2018 at 11:08 AM, David Arnold <dar@xoe.solutions> wrote:
>
> > >This would appear to solve multiline issues within Fluent.....
> > >https://docs.fluentd.org/v0.12/articles/parser_multiline
> >
> > I definitely looked at that, but what guarantees do I have that the
> > sequence is always ERROR/STATEMENT/DETAIL? And not the other way round?
>
> Have you asked that question? You seem to at least have opened the source
> code - did you try to figure out what the logging format is?
I looked at this a couple of days ago. I think parsing with this
library is possible to a certain extent, and the problems stem from
limitations of the library. So, it turns out that the firstline can be
set to a single regex that searches for PANIC, FATAL, ERROR, WARNING,
LOG, NOTICE, DEBUG. That's always the first line in any postgres log
event.
A log event contains some subsequent lines. Those start either with a
tab (which is a continuation of the previous line) or with one of
DETAIL, CONTEXT, HINT, STATEMENT, QUERY. This seems very simple to
parse (just add lineN patterns for those), *except* that the messages
can be multiline too; and where would you assign the continuation lines
for each of those? parser_multiline does not support that.
Another thing worth keeping in mind is that you need to change the regex
depending on log_line_prefix, which sounds very painful.
All in all, the best approach might be to create a specific
parser_postgresql.rb plugin. Seems much easier to write two dozen lines
of Ruby than change all of PostgreSQL's logging infrastructure.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services