Обсуждение: Proposal: Adding json logging

On Sat, Apr 14, 2018 at 12:00:16AM +0000, David Arnold wrote:
> I'm new here. I'm David and would describe myself as an ambitious newbie,
> so please take my suggestion with a grain of salt.

Welcome here.

> I asked Michael Paquier about his solution:
> https://github.com/michaelpq/pg_plugins/tree/master/jsonlog
> He was suggestion to take action and propose this extension again to be
> included in contrib:
> https://github.com/michaelpq/pg_plugins/issues/24
>
> He mentioned the argument was rised of taking too much place.
> This is true under the paradigm that logs are consumed by TTY or grep,
> however, if those logs are to be stored in a logging solution, this is not
> really of concern.

Here are the exact same words I used on this github thread to avoid
confusion:
"I proposed that a couple of years back, to be rejected as the key names
are too much repetitive and take too much place. I have personally plans
to work on other things, so if anybody wishes to take this code and send
a proposal upstream, feel free to! The code is under PostgreSQL license
and I am fine if a patch is proposed even with this code taken."

I am not sure that the concerns expressed back on community-side have
changed.  I cannot put back my finger on the -hackers thread where this
has been discussed by the way, the extra log volume caused by repetitive
key names was one.

> Please let me know if you need more context on my use case.
>
> That beeing said the proposal is to accept this library into postgres
> contrib.
>
> Please let me know, if I should prepare a patch.

It is better to gather opinions before delivering a patch.  If there is
consensus that people would like to have an in-core option to allow logs
in json format, for which I am sure that folks would *not* want a
contrib/ plugin but something as an extension of log_destination, then
of course you could move ahead and propose a patch.  Of course feel free
to reuse any code in my module if that helps!  It is released under
PostgreSQL license as well.

Please note two things though:
- Patch submission follows a particular flow, be sure to read those
notes:
https://wiki.postgresql.org/wiki/Submitting_a_Patch
- Once you have a patch, you need to send it to a commit fest, for which
the next one will likely be next September (precise schedule will be
finalized at the end of May at PGCon) for the beginning of development
of Postgres 12.  The development of Postgres 11 has just finished, so
the focus is to stabilize the release first, which consists in testing
and double-checking that everything which has been merged is stable.  So
please do not expect immediate feedback on any patch you send.

Thanks,
--
Michael

On 14 April 2018 at 11:24, Michael Paquier <michael@paquier.xyz> wrote:

> "I proposed that a couple of years back, to be rejected as the key names
> are too much repetitive and take too much place.

gzip is astonishingly good at dealing with that, so I think that's
actually a bit of a silly reason to block it.

Plus it's likely only a short-lived interchange format, not something
to be retained for a long period.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

On Sat, Apr 14, 2018 at 03:27:58PM +0000, David Arnold wrote:
> > Plus it's likely only a short-lived interchange format, not something to be
> retained for a long period.
> 
> Absolutely.
> 
> There might be an argument that it's not easy on the eyes in the case it
> would be consumed by a pair of them. It's absolutely valid. Golang
> community has found a solution for that called logfmt, which I personally
> appreciate.

I think a suite of json_to_* utilities would be a good bit more
helpful in this regard than changing our human-eye-consumable logs. We
already have human-eye-consumable logs by default.  What we don't
have, and increasingly do want, is a log format that's really easy on
machines.

As to logfmt in particular, the fact that it's not standardized is
probably a show-stopper.

Let's go with JSON.

Best,
David.
-- 
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter <david@fetter.org> writes:
> I think a suite of json_to_* utilities would be a good bit more
> helpful in this regard than changing our human-eye-consumable logs. We
> already have human-eye-consumable logs by default.  What we don't
> have, and increasingly do want, is a log format that's really easy on
> machines.

I'm dubious that JSON is "easier on machines" than CSV.

            regards, tom lane

On Sat, Apr 14, 2018 at 11:51:17AM -0400, Tom Lane wrote:
> David Fetter <david@fetter.org> writes:
> > I think a suite of json_to_* utilities would be a good bit more
> > helpful in this regard than changing our human-eye-consumable
> > logs. We already have human-eye-consumable logs by default.  What
> > we don't have, and increasingly do want, is a log format that's
> > really easy on machines.
> 
> I'm dubious that JSON is "easier on machines" than CSV.

I've found the opposite.

CSV is very poorly specified, which makes it at best complicated to
build correct parsing libraries. JSON, whatever gripes I have about
the format[1] is extremely well specified, and hence has excellent
parsing libraries.

Best,
David.

[1] These are mostly the lack of comments and of some useful data
types like large integers, floats, and ISO-8601 dates.  PostgreSQL
continues to share that last.
-- 
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On 2018-04-14 18:05:18 +0200, David Fetter wrote:
> On Sat, Apr 14, 2018 at 11:51:17AM -0400, Tom Lane wrote:
> > David Fetter <david@fetter.org> writes:
> > > I think a suite of json_to_* utilities would be a good bit more
> > > helpful in this regard than changing our human-eye-consumable
> > > logs. We already have human-eye-consumable logs by default.  What
> > > we don't have, and increasingly do want, is a log format that's
> > > really easy on machines.
> > 
> > I'm dubious that JSON is "easier on machines" than CSV.
> 
> I've found the opposite.
> 
> CSV is very poorly specified, which makes it at best complicated to
> build correct parsing libraries. JSON, whatever gripes I have about
> the format[1] is extremely well specified, and hence has excellent
> parsing libraries.

Worth to notice that useful json formats for logging also kinda don't
follow standards. Either you end up with entire logfiles as one big
array, which most libraries won't parse and makes logrotate etc really
complicated, or you end up with some easy to parse format where newlines
have non-standard record separator meaning.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> On 2018-04-14 18:05:18 +0200, David Fetter wrote:
>> CSV is very poorly specified, which makes it at best complicated to
>> build correct parsing libraries. JSON, whatever gripes I have about
>> the format[1] is extremely well specified, and hence has excellent
>> parsing libraries.

> Worth to notice that useful json formats for logging also kinda don't
> follow standards. Either you end up with entire logfiles as one big
> array, which most libraries won't parse and makes logrotate etc really
> complicated, or you end up with some easy to parse format where newlines
> have non-standard record separator meaning.

Hmm .. that, actually, seems like a pretty serious objection.  If the beef
with CSV is that it's poorly specified and inconsistently implemented
(which is surely true), then using some nonstandard variant of JSON
doesn't seem like it's going to lead to a big step forward.

"The wonderful thing about standards is there are so many to choose from."
(variously attributed to Hopper, Tanenbaum, and others)

            regards, tom lane

On Sat, Apr 14, 2018 at 01:20:16PM -0700, Andres Freund wrote:
> On 2018-04-14 18:05:18 +0200, David Fetter wrote:
> > On Sat, Apr 14, 2018 at 11:51:17AM -0400, Tom Lane wrote:
> > > David Fetter <david@fetter.org> writes:
> > > > I think a suite of json_to_* utilities would be a good bit more
> > > > helpful in this regard than changing our human-eye-consumable
> > > > logs. We already have human-eye-consumable logs by default.  What
> > > > we don't have, and increasingly do want, is a log format that's
> > > > really easy on machines.
> > > 
> > > I'm dubious that JSON is "easier on machines" than CSV.
> > 
> > I've found the opposite.
> > 
> > CSV is very poorly specified, which makes it at best complicated to
> > build correct parsing libraries. JSON, whatever gripes I have about
> > the format[1] is extremely well specified, and hence has excellent
> > parsing libraries.
> 
> Worth to notice that useful json formats for logging also kinda don't
> follow standards. Either you end up with entire logfiles as one big
> array, which most libraries won't parse and makes logrotate etc really
> complicated, or you end up with some easy to parse format where newlines
> have non-standard record separator meaning.

I don't see this as a big problem.  The smallest-lift thing is to put
something along the lines of:

    When you log as JSON, those logs are JSON objects, one per output
    event.  They are not guaranteed to break on newlines.

A slightly larger lift would include escaping newlines and ensuring
that JSON output is always single lines, however long.

Best,
David.
-- 
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On 2018-04-15 00:31:14 +0200, David Fetter wrote:
> On Sat, Apr 14, 2018 at 01:20:16PM -0700, Andres Freund wrote:
> > On 2018-04-14 18:05:18 +0200, David Fetter wrote:
> > > CSV is very poorly specified, which makes it at best complicated to
> > > build correct parsing libraries. JSON, whatever gripes I have about
> > > the format[1] is extremely well specified, and hence has excellent
> > > parsing libraries.
> > 
> > Worth to notice that useful json formats for logging also kinda don't
> > follow standards. Either you end up with entire logfiles as one big
> > array, which most libraries won't parse and makes logrotate etc really
> > complicated, or you end up with some easy to parse format where newlines
> > have non-standard record separator meaning.
> 
> I don't see this as a big problem.  The smallest-lift thing is to put
> something along the lines of:
> 
>     When you log as JSON, those logs are JSON objects, one per output
>     event.  They are not guaranteed to break on newlines.
> 
> A slightly larger lift would include escaping newlines and ensuring
> that JSON output is always single lines, however long.

Still obliterates your "standard standard standard" line of
argument. There seem to valid arguments for adding json regardless, but
that line is just bogus.

Greetings,

Andres Freund

I would suggest that the community consider whether postgres will log multidimensional data. That will weigh into the
decisionof  json vs. another format quite significantly. I am a fan of the json5 spec (https://json5.org/), though
adoptionof this is quite poor. 
 


---
Jordan Deitch
https://id.rsa.pub

> > I would suggest that the community consider whether postgres will log
> multidimensional data. That will weigh into the decision of  json vs.
> another format quite significantly. I am a fan of the json5 spec (
> https://json5.org/), though adoption of this is quite poor.
> 
> What do you mean by multidimensional data? Arrays/maps?
> 
> I think there is no advantage of multidimensional vs prefixed flat logging
> unless data structure gets really nastily nested.
> 
> What case where you thinking of?

Exactly - arrays, maps, nested json objects. It's more organized and easier to reason about. As postgresql becomes more
andmore sophisticated over time, I see flat logging becoming more unwieldy. With tools like jq, reading and querying
jsonon the command line is simple and user friendly, and using json for logging capture and aggregation is widely
supportingand embraced.
 

> Exactly what are you logging here ??? Why would I need to see a
> multi-dimensional array in the log ?

If I wanted to capture the location of errors my clients are encountering on their postgres clusters in detail, I would
needto parse the 'LOCATION' string in their log entries, parse out the filename by splitting on the ':' character of
thatsame line, and parse out the line number. Essentially any programmatic analysis of logs, as it stands today, would
requirestring parsing. I'd rather have an organized, logical representation of information which I suggest is not
possiblein a flat, single dimensional structure. 
 

{
"level":"ERROR",
"meta":{
 "line_number":23,
 "file": "parse_relation.c",
},
"detail:{
 "condition_name"...,
 "error_code"...,
},
time:....
}

> On Apr 15, 2018, at 09:51, David Arnold <dar@xoe.solutions> wrote:
>
> 1. Throughout this vivid discussion a good portion of support has already been manifested for the need of a more
structured(machine readable) logging format. There has been no substantial objection to this need. 

I'm afraid I don't see that.  While it's true that as a standard, CSV is relatively ill-defined, as a practical matter
inPostgreSQL it is very easy to write code that parses .csv format. 

--
-- Christophe Pettus
   xof@thebuild.com

> On Apr 15, 2018, at 10:07, Christophe Pettus <xof@thebuild.com> wrote:
>
>
>> On Apr 15, 2018, at 09:51, David Arnold <dar@xoe.solutions> wrote:
>>
>> 1. Throughout this vivid discussion a good portion of support has already been manifested for the need of a more
structured(machine readable) logging format. There has been no substantial objection to this need. 
>
> I'm afraid I don't see that.  While it's true that as a standard, CSV is relatively ill-defined, as a practical
matterin PostgreSQL it is very easy to write code that parses .csv format. 

More specifically, JSON logging does seem to be a solution in search of a problem.  PostgreSQL's CSV logs are very easy
tomachine-parse, and if there are corrupt lines being emitted there, the first step should be to fix those, rather than
introducea new "this time, for sure" logging method. 

It's a matter of a few lines of code to convert CSV logs to a JSON format, if you need JSON format for something else.

Remember, also, that every new logging format introduces a burden on downstream tools to support it.  This is (still)
anissue with JSON format plans, which had a much more compelling advantage over standard-format plans than JSON logs do
overCSV. 

--
-- Christophe Pettus
   xof@thebuild.com

> On Apr 15, 2018, at 10:39, David Arnold <dar@xoe.solutions> wrote:
>
> In the light of the specific use case / problem for this thread to be born, what exactly would you suggest?

It looks like the thread skipped over the problem space for the solution space pretty fast; I see your note:

> I have reviewed some log samples and all DO contain some kind of multi line logs which are very uncomfortable to
parsereliably in a log streamer. 

... but I don't see any actual examples of those.  Can you elaborate?

--
-- Christophe Pettus
   xof@thebuild.com

On 2018-04-15 18:00:05 +0000, David Arnold wrote:
> CSV shows line breaks, STDOUT shows ERROR/FATAL and detail on different
> lines, not an easy problem to stream-parse reliably (without some kind of a
> buffer, etc)...

Why? The newlines aren't meaningfully different from other characters
you need to parse? The data isn't actually stored in a newline separated
fashion, that's just one byte with that meaning.

- Andres

> On Apr 15, 2018, at 11:00, David Arnold <dar@xoe.solutions> wrote:
> 
> CSV Logs: https://pastebin.com/uwfmRdU7

Is the issue that there are line breaks in things like lines 7-9?

--
-- Christophe Pettus
   xof@thebuild.com

> On Apr 15, 2018, at 12:16, David Arnold <dar@xoe.solutions> wrote:
>
> Core-Problem: "Multi line logs are unnecessarily inconvenient to parse and are not compatible with the design of some
(commonlyused) logging aggregation flows." 

I'd argue that the first line of attack on that should be to explain to those consumers of logs that they are making
someunwarranted assumptions about the kind of inputs they'll be seeing.  PostgreSQL's CSV log formats are not a
particularbizarre format, or very difficult to parse.  The standard Python CSV library handles them just file, for
example. You have to handle newlines that are part of a log message somehow; a newline in a PostgreSQL query, for
example,needs to be emitted to the logs. 

--
-- Christophe Pettus
   xof@thebuild.com

On 04/15/2018 05:05 PM, Christophe Pettus wrote:
>> On Apr 15, 2018, at 12:16, David Arnold <dar@xoe.solutions> wrote:
>>
>> Core-Problem: "Multi line logs are unnecessarily inconvenient to parse and are not compatible with the design of
some(commonly used) logging aggregation flows."
 
> I'd argue that the first line of attack on that should be to explain to those consumers of logs that they are making
someunwarranted assumptions about the kind of inputs they'll be seeing.  PostgreSQL's CSV log formats are not a
particularbizarre format, or very difficult to parse.  The standard Python CSV library handles them just file, for
example. You have to handle newlines that are part of a log message somehow; a newline in a PostgreSQL query, for
example,needs to be emitted to the logs.
 
>


In JSON newlines would have to be escaped, since literal newlines are
not legal in JSON strings. Postgres' own CSV parser has no difficulty at
all in handling newlines embedded in the fields of CSV logs.

I'm not necessarily opposed to providing for JSON logs, but the overhead
of named keys could get substantial. Abbreviated keys might help, but
generally I think I would want to put such logs on a compressed ZFS
drive or some such.

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Apr 16, 2018 at 10:06:29AM -0400, Andrew Dunstan wrote:
> On 04/15/2018 05:05 PM, Christophe Pettus wrote:
> >> On Apr 15, 2018, at 12:16, David Arnold <dar@xoe.solutions> wrote:
> >>
> >> Core-Problem: "Multi line logs are unnecessarily inconvenient to parse and are not compatible with the design of
some(commonly used) logging aggregation flows."
 
> > I'd argue that the first line of attack on that should be to explain to those consumers of logs that they are
makingsome unwarranted assumptions about the kind of inputs they'll be seeing.  PostgreSQL's CSV log formats are not a
particularbizarre format, or very difficult to parse.  The standard Python CSV library handles them just file, for
example. You have to handle newlines that are part of a log message somehow; a newline in a PostgreSQL query, for
example,needs to be emitted to the logs.
 
> 
> 
> In JSON newlines would have to be escaped, since literal newlines are
> not legal in JSON strings. Postgres' own CSV parser has no difficulty at
> all in handling newlines embedded in the fields of CSV logs.

True, and anything that malloc()s in the process of doing that
escaping could fail on OOM, and hilarity would ensue. I don't see
these as show-stoppers, or even as super relevant to the vast majority
of users. If you're that close to the edge, you were going to crash
anyhow.

> I'm not necessarily opposed to providing for JSON logs, but the
> overhead of named keys could get substantial. Abbreviated keys might
> help, but generally I think I would want to put such logs on a
> compressed ZFS drive or some such.

Frequently at places I've worked, the end destination is of less
concern immediate than the ability to process those logs for
near-real-time monitoring.  This is where formats like JSON really
shine.

Best,
David.
-- 
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

    David Arnold wrote:

> Not claiming this assumption does imply parsing of a *rolling* set
> of log lines with *previously unkown cardinality*. That's expensive
> on computing resources. I don't have actual numbers, but it doesn't
> seem too far fetched, neither.
> I filed a question to the author of fluent-bit to that extend which
> you can consult here:
> https://github.com/fluent/fluent-bit/issues/564 Let's see what
> Eduardo has to inform us about this...

fluent-bit does not appear to support CSV, as mentioned in
https://github.com/fluent/fluent-bit/issues/459
which got flagged as an enhancement request some time ago.

In CSV a line break inside a field is easy to process for
a parser, because (per https://tools.ietf.org/html/rfc4180):

  "Fields containing line breaks (CRLF), double quotes, and commas
    should be enclosed in double-quotes"

So there is no look ahead to do. In a character-by-character loop,
when encountering a line break, either the current field did not
start with a double quote and the line break is part of the content, or it
did
start with a double quote and the line break ends the current record.

What doesn't quite work is to parse CSV with a regex, it's
discussed in some detail here for instance:
https://softwareengineering.stackexchange.com/questions/166454/can-the-csv-format-be-defined-by-a-regex


Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite

On 4/13/18 20:00, David Arnold wrote:
> I have reviewed some log samples and all DO contain some kind of multi
> line logs which are very uncomfortable to parse reliably in a log streamer.
> 
> I asked Michael Paquier about his
> solution: https://github.com/michaelpq/pg_plugins/tree/master/jsonlog
> He was suggestion to take action and propose this extension again to be
> included in contrib:
> https://github.com/michaelpq/pg_plugins/issues/24 

I have used https://github.com/mpihlak/pg_logforward in the past, which
seems to be about the same thing.

I have also had good success using syslog.  While syslog is not very
structured, the setting syslog_split_messages allows sending log entries
that include newlines in one piece, which works well if you have some
kind of full-text search engine at the receiving end.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Apr 16, 2018 at 07:52:58PM -0400, Peter Eisentraut wrote:
> I have used https://github.com/mpihlak/pg_logforward in the past, which
> seems to be about the same thing.

Didn't know this one.  Thanks.

> I have also had good success using syslog.  While syslog is not very
> structured, the setting syslog_split_messages allows sending log entries
> that include newlines in one piece, which works well if you have some
> kind of full-text search engine at the receiving end.

syslog suffers from the possibility to lose messages if I recall
correctly, right?  This may matter for some critical environments.

Upstream code has escape_json() directly included, which is able to do
the job and makes sure that a single JSON entry is not broken into
multiple lines.  That's what my jsonlog uses to format the strings used,
and what I can see pg_logforward does as well witha custom copy.

As a whole model, producing one JSON object per line and per log-entry
is the most natural format in my opinion.

One thing which is perhaps sensitive for JSON is the timestamp format.
The JSON specification does not decide what should be the format of
timestamps, still parser facilities are somewhat all pointing into using
ISO 8601 with stuff like Javascript Date's toJSON method.  There are
some side issues with the use of UTC..  So the thing is sorta of messy.

However, as JSON entries are usually larger than normal log entries,
getting log entries broken into multiple lines is easier if not using
logging_collector.  People normally don't do that, but I received
complains on the matter as well when using Postgres in Docker container
for example.  So documenting that logging_collector needs to be enabled
is important if this log format shows up in Postgres.
--
Michael

On 4/16/18 23:12, Michael Paquier wrote:
>> I have also had good success using syslog.  While syslog is not very
>> structured, the setting syslog_split_messages allows sending log entries
>> that include newlines in one piece, which works well if you have some
>> kind of full-text search engine at the receiving end.
> syslog suffers from the possibility to lose messages if I recall
> correctly, right?  This may matter for some critical environments.

Depends on whether you configure it to use TCP or UDP.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

    David Arnold wrote:

> Interesting, does that implicitly mean the whole log event would get
> transmitted as a "line" (with CRLF) in CSV.

To me it's implied by the doc at:
https://www.postgresql.org/docs/current/static/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-CSVLOG

> In the affirmative scenario, this then would work for a true streaming
> aggregator (if CSV was supported).

Assuming a real CSV parser tailing the log, there shouldn't be any trouble
with that.


Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite

One issue I haven't seen mentioned in this thread is the translation
status of the server message (as well as its encoding): it's possible to
receive messages in some random language if the lc_message setting is
changed.  Requiring that lc_messages must always be set to some English
locale seems like a poor answer to this problem.
IMO the untranslated server message should be part of the event also.
I don't know what to think of %-expansions of the message.

The character encoding can be changed per database.  Log files where the
encoding differs across databases cannot be processed in any sane way.
You can try some heuristics (try to read each message as utf8 first, and
if that fails, then it must be Latin1!  If that doesn't work for you,
... tough luck), but that's a pretty poor answer too.  Not sure what is
a good solution to this problem.  Maybe ensure that these things are
always UTF8?

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sun, Apr 15, 2018 at 1:07 PM, Christophe Pettus <xof@thebuild.com> wrote:
>> On Apr 15, 2018, at 09:51, David Arnold <dar@xoe.solutions> wrote:
>> 1. Throughout this vivid discussion a good portion of support has already been manifested for the need of a more
structured(machine readable) logging format. There has been no substantial objection to this need.
 
>
> I'm afraid I don't see that.  While it's true that as a standard, CSV is relatively ill-defined, as a practical
matterin PostgreSQL it is very easy to write code that parses .csv format.
 

I'm not sure exactly how you intended to this comment, but it seems to
me that whether CSV is ease or hard to parse, somebody might
legitimately find JSON more convenient.  For example, and as has been
discussed on this thread, if you have a system that is consuming the
logs that already knows how to parse JSON but does not know how to
parse CSV, then you will find the JSON format to be convenient.

For the record, I'm tentatively in favor of including something like
this in contrib.  I think it's useful to have more examples of how to
use our existing hooks in contrib, and I think this is useful on
principle.

I am a little concerned about this bit from the README, though:

====
Note that logging_collector should be enabled in postgresql.conf to
ensure consistent log outputs.  As JSON strings are longer than normal
logs generated by PostgreSQL, this module increases the odds of malformed
log entries.
====

I'm not sure I understand the issue, but I don't like malformed log entries.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

> On Apr 18, 2018, at 11:59, Robert Haas <robertmhaas@gmail.com> wrote:
>
> I'm not sure exactly how you intended to this comment, but it seems to
> me that whether CSV is ease or hard to parse, somebody might
> legitimately find JSON more convenient.

Of course.  The specific comment I was replying to made a couple of jumps that I wanted to unwind: The first is that we
don'thave a machine-readable format for PostgreSQL (we do, CSV), and that there was "no substantial objection to this
need."

If the requirement is: "There is a large class of log analysis tool out there that has trouble with multiline formats
andwe should be good ecosystem players," that's fine.  (I'm a bit sour about the number of tools being written with
one-line-per-eventbaked into them and whose solution to any other format is "use regex," but that's neither here nor
there,I suppose.) 

My primary objection to creating new output formats is that it creates an implicit burden on downstream tools to adopt
them. For example, a log of query analysis tools don't yet process JSON-format plans, and they've been around for a
while. By introducing a new format in core (which was the starting proposal), we're essentially telling all the tools
(suchas pgbadger) that might absorb them that we expect them to adopt that too. 

> For the record, I'm tentatively in favor of including something like
> this in contrib.

I'm much less fussed by this in contrib/ (with the same concern you noted), at minimum as an example of how to do
loggingin other formats. 

--
-- Christophe Pettus
   xof@thebuild.com

John W Higgins wrote:
> On Sun, Apr 15, 2018 at 11:08 AM, David Arnold <dar@xoe.solutions> wrote:
> 
> > >This would appear to solve multiline issues within Fluent.....
> > >https://docs.fluentd.org/v0.12/articles/parser_multiline
> >
> > I definitely looked at that, but what guarantees do I have that the
> > sequence is always ERROR/STATEMENT/DETAIL? And not the other way round?
> 
> Have you asked that question? You seem to at least have opened the source
> code - did you try to figure out what the logging format is?

I looked at this a couple of days ago.  I think parsing with this
library is possible to a certain extent, and the problems stem from
limitations of the library.  So, it turns out that the firstline can be
set to a single regex that searches for PANIC, FATAL, ERROR, WARNING,
LOG, NOTICE, DEBUG.  That's always the first line in any postgres log
event.

A log event contains some subsequent lines.  Those start either with a
tab (which is a continuation of the previous line) or with one of
DETAIL, CONTEXT, HINT, STATEMENT, QUERY.  This seems very simple to
parse (just add lineN patterns for those), *except* that the messages
can be multiline too; and where would you assign the continuation lines
for each of those?  parser_multiline does not support that.

Another thing worth keeping in mind is that you need to change the regex
depending on log_line_prefix, which sounds very painful.

All in all, the best approach might be to create a specific
parser_postgresql.rb plugin.  Seems much easier to write two dozen lines
of Ruby than change all of PostgreSQL's logging infrastructure.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Wed, Apr 18, 2018 at 02:59:26PM -0400, Robert Haas wrote:
> ====
> Note that logging_collector should be enabled in postgresql.conf to
> ensure consistent log outputs.  As JSON strings are longer than normal
> logs generated by PostgreSQL, this module increases the odds of malformed
> log entries.
> ====
>
> I'm not sure I understand the issue, but I don't like malformed log entries.

If logging_collector is disabled, then all the log entries would just go
to stderr (that's mentioned in the docs).  While that may be fine for
low volumes of logs, for many concurrent processes generating logs then
a process can overwrite another process log entry.  The logs generated by
the JSON format are longer in length, increased mainly by the repetitive
use of the key names in the blob, which increase in turn the volume
generated.

So in this case this can cause JSON blobs to look broken.  I had a
report on github not long ago about that:
https://github.com/michaelpq/pg_plugins/issues/17
--
Michael

On Wed, Apr 18, 2018 at 12:10:47PM -0700, Christophe Pettus wrote:
> On Apr 18, 2018, at 11:59, Robert Haas <robertmhaas@gmail.com> wrote:
>> For the record, I'm tentatively in favor of including something like
>> this in contrib.
>
> I'm much less fussed by this in contrib/ (with the same concern you
> noted), at minimum as an example of how to do logging in other
> formats.

Using a contrib module for logging format has also a side effect. When
the logging collector is disabled, all the log entries which are created
by the postmaster have junk data as it is sort of impossible to make the
loaded module know that the logging collector is enabled in
configuration but that the log entries cannot use the pipe protocol
yet.  In short, you finish with a couple of entries which are formatted
for the pipe protocol used by the syslogger but are redirected to
stderr.  There are only a couple of entries which enter in this
category, like a misconfiguration of the server, or the ports the server
is listening to (look for redirection_done in elog.c).  One simple fix
would be to pass down the value of redirection_done to emit_log_hook,
and this requires patching the server.
--
Michael