Improve logging performance

[yomed]

In cases with lots of logging and expensive logging functions, performance can suffer significantly. We should consider a few ways of improving performance for these cases. For example:

1. Collect aggregate metrics about how often warnings are generated, and then log only those metrics (rather that the atomic data for each transaction).
2. Only log incremental data when the aggregate error rate is high.
3. Flush all of the logs at the same time rather than incrementally. Note that this does make assumptions about the underlying logging systems which are independent of this module.

[sidsakhadeo]

Seems like an improvement we should consider. Regarding your points:

1. Collect aggregate metrics about how often warnings are generated, and then log only those metrics (rather that the atomic data for each transaction).

Are you suggesting, there be a feature where the warnings are not logged and only the total count of these warnings are published?

2. Only log incremental data when the aggregate error rate is high.

Can you provide an example of how this would work?

3. Flush all of the logs at the same time rather than incrementally. Note that this does make assumptions about the underlying logging systems which are independent of this module.

This is really useful I agree. We can provide an option to do batch logging and the user can opt into using it should their logging system provide it.

[yomed]

Right, I think these might be able to work in tandem.

1 might be a good option for production environment, and would log:

Warnings: 30, DATA_MISSING: 20, TYPE_MISMATCH: 10

I think 2+3 should probably work together. I.e. store all the incremental logs and flush them at the end (maybe only log if them if there are x warnings):

Warnings: 30, DATA_MISSING: 20, TYPE_MISMATCH: 10
Warning 1: ...
Warning 2: ...
...

For all these cases I think the user would have to specify something like r.startLogging() and r.stopLogging() so we would know how to manage the aggregation and flushing.