Re: Proposal: Adding json logging - Mailing list pgsql-hackers
> I'd argue that the first line of attack on that should be to explain to those consumers of logs that they are making some unwarranted assumptions about the kind of inputs they'll be seeing. PostgreSQL's CSV log formats are not a particular bizarre format, or very difficult to parse. The standard Python CSV library handles them just file, for example. You have to handle newlines that are part of a log message somehow; a newline in a PostgreSQL query, for example, needs to be emitted to the logs.
I believe the (valid) root reason behind this assumption is to care for resource consumption during stream processing. There are solutions to that problem (as the multi line mode of fuent bit already exposed before in this thread), but they are not specifically reliable, nor easy to maintain. Not claiming this assumption does imply parsing of a rolling set of log lines with previously unkown cardinality. That's expensive on computing resources. I don't have actual numbers, but it doesn't seem too far fetched, neither.
I filed a question to the author of fluent-bit to that extend which you can consult here: https://github.com/fluent/fluent-bit/issues/564
Let's see what Eduardo has to inform us about this...
El dom., 15 abr. 2018 a las 16:05, Christophe Pettus (<xof@thebuild.com>) escribió:
> On Apr 15, 2018, at 12:16, David Arnold <dar@xoe.solutions> wrote:
>
> Core-Problem: "Multi line logs are unnecessarily inconvenient to parse and are not compatible with the design of some (commonly used) logging aggregation flows."
I'd argue that the first line of attack on that should be to explain to those consumers of logs that they are making some unwarranted assumptions about the kind of inputs they'll be seeing. PostgreSQL's CSV log formats are not a particular bizarre format, or very difficult to parse. The standard Python CSV library handles them just file, for example. You have to handle newlines that are part of a log message somehow; a newline in a PostgreSQL query, for example, needs to be emitted to the logs.
--
-- Christophe Pettus
xof@thebuild.com
 | DAVID ARNOLD Gerente General |
| xoe.solutions dar@xoe.solutions +57 (315) 304 13 68 | |
| Confidentiality Note: This email may contain confidential and/or private information. If you received this email in error please delete and notify sender. | |
| Environmental Consideration: Please avoid printing this email on paper, unless really necessary. | |
pgsql-hackers by date: