log4j -adaptive log level- feature

Don't want to reinvent the wheel so I am wondering if any logging system already supports something like what I am proposing to do.

Background: I am working on an extremely large system where tens of thousands of users access the servers at any given time. There are a lot of surrounding infrastructures involved so you can imagine what it looks like to investigate rare occurring bugs just by reading the logs in such an ecosystem.

Our system uses log4j.

The main root of the problem was that the original implementors were rather 'economical', to put it mildly, when encountering an unknown error. Sure, the DEBUG level would be helpful in most cases, but of course, production logs are set on ERROR level only :(

Now I am trying to make a better world for our children and want to expand the logging system to be more 'forgiving' about the log level.

What I have in mind goes like this:
Since the surrounding DEBUG level logs (N before and M after the ERROR log) might contain important data, why not lower the log level of the system around the ERROR.

My idea:

  1. let's assume ERROR log level set for the system
  2. use some kind of a rollover logging buffer of size X. Every log goes in there and is forwarded to log4j.
  3. When buffer gets an ERROR log it can also output the context of N previous and M next log messages by either
    1. artificially raising the log level to ERROR state (and noting that, of course) or
    2. just log that context via an injected ERROR log

There are some issues to work out but in general it could work.

Any ideas?

Cheers

6
задан Charles 31 December 2011 в 20:56
поделиться