Introduce logging framework

[frank-bee]

Currently errors are thrown as follows: throw Error("rocksdb open failed: " + err);
Like this they end up on standard error (logs to standard out).
This is very difficult to parse and to used e.g. for alertings.

Let's introduce a log framework, e.g.
https://github.com/pinojs/pino

see also #239

[frank-bee]

One advantage of the introduction of a logging framework would be:
We could get rid of the email notifications, instead it would be up the hoster of a node to configure alerts with the configured logging / alerting backend ( e.g. AWS cloudwatch / SNS)

[tonyofbyteball]

Agreed that some more advanced logging that would allow to specify and filter by log level would make sense.

I took a quick look at pino and not sure having logs in json is a good idea, that would make their manual inspection without additional tools more difficult.

Not sure I get your idea of getting rid of email notifications, what for? Are you suggesting something hoster-specific?

[tarmo888]

There are some other cloud solutions for logging too, but these get very costly with the amount of logs that Obyte nodes generate https://www.papertrail.com/

As I understand, @frank-bee suggestion is to replace those email notifications with some other logging system, so there would be no need to have sendmail or SMTP.

[tonyofbyteball]

Other things being equal, I would always prefer standard, well-known, and widely available email over third-party services with their proprietary APIs.

[frank-bee]

advanced logging that would allow to specify and filter by log level would make sense.

That one of the advantages. Furthermore it allows you to react differently on different types of logs, e.g. for error or fatal errors one could inform some admin via alert.

In general it makes sense to introduce some kind of logging architecture. It is a way of doing "seperation of concerns" which are here: collection, transport, storage, analysis, alerting.
The only thing to be done in the app code is the collection. All the other concerns would be done in another process and by standard tools. you may ask why?
Some advantages of such a setup are

• transport should be done by a different process for performance and security reasons (e.g. if app crashes , it cannot sent the last and most important error messages
• storage could be done on a suitable medium (e.g. soemthing like cloudwatch in AWS or an ELK cluster). on the actual node storage might be limited.
• Analysis might be optional, no need in each setup
• Alerting could be done in different ways, again depending on the usecase (e.g. some people might have a slack channel for the operations teams which should be alerted, others want to use email

[frank-bee]

There are some other cloud solutions for logging too, but these get very costly with the amount of logs that Obyte nodes generate https://www.papertrail.com/

As I understand, @frank-bee suggestion is to replace those email notifications with some other logging system, so there would be no need to have sendmail or SMTP.

I would not replace it but rather get rid of it. If you have the logs in a proper format, it is up the node hoster to "do someting with the logs". As I described about, typically your store the logs somewhere for later analysis ( e.g. papertrail, cloudwatch,...) plus you confiugre some alarm on e.g. errors (e.g cloudwatch + SNS in AWS or something similar in ELK or papertrail)

[frank-bee]

I took a quick look at pino and not sure having logs in json is a good idea, that would make their manual inspection without additional tools more difficult.

https://getpino.io/#/docs/pretty

[tarmo888]

Not sure I get it.

Everything, which has the headless-obyte as dependency is already writing to log file, so if you mount ~/.config/headless-obyte/ do different drive then you can have the node drive separated from the data drive. And even without mounting to separate, you could make something to tail the log file from main drive.

Everything without headless-obyte, you can still output the stdout and stderr to logfile and tail that log while with some other process. It's all doable already today, just matter of how you want to set it up.

[frank-bee]

Currently the main issues I see regarding logging:

• no standard log format with timestamp (see my orignial post above)
• email feature ( I cannot use an smtp server in my setup but still want to configure alerts)

I don't think there is much to implement but basically agree on a way to do logging ( to some extend)

[tarmo888]

What is missing in the log.txt that gets generated with headless-obyte or from the screen of other nodes that you can direct to any file? I see that each line starts with:

Thu Mar 12 2020 19:13:27 GMT+0000 (Coordinated Universal Time): SOMETHING