Re: Proposal: Adding json logging - Mailing list pgsql-hackers
This discussion is thriving, and long and behold, we've got an opinion from Eduardo (fluent-bit):
https://github.com/fluent/fluent-bit/issues/564#issuecomment-381844419
>Also consider that in not all scenarios full multiline logs are flushed right away, sometimes there are delays.
I think this line supports the view for multi line being a sub optimal logging strategy (assessed globally and embedded into an deployment ecosystem).
> Btw, we would be happy to work together with PostreSQL community to support an official parser for your multiline logs
Although, fluent-bit seem to strive to work with postgres log "as it pleases", I think it is still a valid problem definition against all contrary arguments to maintain:
Core-Problem: "Multi line logs are unnecessarily inconvenient to parse and are not compatible with the design of some (commonly used) logging aggregation flows."
2nd-order Problem: "Logging space increasingly moves towards the adoption of structured logging formats around json/logfmt. Compatibly options (plural!) with main stream (not necessarily standard) tooling is a value proposition of it's own kind. It helps increase odds of responsible deployments and improves the overall experience in adopting PostgreSQL."
Please share you thoughts, if you still feel there are material objections to the core problem? JSON or not JSON, as Christophe recalled, then is a question in the solution space.
https://github.com/fluent/fluent-bit/issues/564#issuecomment-381844419
>Also consider that in not all scenarios full multiline logs are flushed right away, sometimes there are delays.
I think this line supports the view for multi line being a sub optimal logging strategy (assessed globally and embedded into an deployment ecosystem).
> Btw, we would be happy to work together with PostreSQL community to support an official parser for your multiline logs
Although, fluent-bit seem to strive to work with postgres log "as it pleases", I think it is still a valid problem definition against all contrary arguments to maintain:
Core-Problem: "Multi line logs are unnecessarily inconvenient to parse and are not compatible with the design of some (commonly used) logging aggregation flows."
2nd-order Problem: "Logging space increasingly moves towards the adoption of structured logging formats around json/logfmt. Compatibly options (plural!) with main stream (not necessarily standard) tooling is a value proposition of it's own kind. It helps increase odds of responsible deployments and improves the overall experience in adopting PostgreSQL."
Please share you thoughts, if you still feel there are material objections to the core problem? JSON or not JSON, as Christophe recalled, then is a question in the solution space.
Note that part of the problem definition is "unnecessary", which implies judgment on responsibilities and ecosystems working together, rather than a broken system.
El mar., 17 abr. 2018 a las 7:31, David Arnold (<dar@xoe.solutions>) escribió:
>To me it's implied by the doc at:
https://www.postgresql.org/docs/current/static/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-CSVLOGAdditionally this still depends on the way some middleware might choose to stream data. Can we really be sure the risk is minimal that any middleware would have chosen to treat new line as an entity delimitator?Can we even be sure that NO existing middleware would treat newline as a entity delimitator?I'm not that confident about that.Anticipating the possible argument, that the "others are wrong": This arguement, though valid, seems sometimes is tought it's limits in very mondane practicability and efficiency needs.El mar., 17 abr. 2018, 6:55 a.m., Daniel Verite <daniel@manitou-mail.org> escribió:David Arnold wrote:
> Interesting, does that implicitly mean the whole log event would get
> transmitted as a "line" (with CRLF) in CSV.
To me it's implied by the doc at:
https://www.postgresql.org/docs/current/static/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-CSVLOG
> In the affirmative scenario, this then would work for a true streaming
> aggregator (if CSV was supported).
Assuming a real CSV parser tailing the log, there shouldn't be any trouble
with that.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
 | DAVID ARNOLD Gerente General |
| xoe.solutions dar@xoe.solutions +57 (315) 304 13 68 | |
| Confidentiality Note: This email may contain confidential and/or private information. If you received this email in error please delete and notify sender. | |
| Environmental Consideration: Please avoid printing this email on paper, unless really necessary. | |
pgsql-hackers by date: