Should I buffer in error logging?

I am using log4net and a customized AdoNetAppender to log error messages into a database.

Is it really a good idea to set the buffer size to 10, 20 or maybe even 100 for logging?

I know that it's good for performance, because this way not every error will trigger a DB query and when many errors occur at once, this may even kill my server, but

It makes immediate error analysis kind of hard, if I have to wait until 99 more errors occur, until I can access the error information in the DB.

Am I missing somethin开发者_开发技巧g? What is the way to go regarding buffering error messages?

Probably the difference is philosophical and some people would say to only do batch inserts with lots of data every now and then.

we actually write immediately to a log file exactly to be able to debug immediately and not have to wait until 99 more errors come.

you can eventually have a mixed approach, to log (text) files immediately and to database every 100 records...

Is it really a good idea to set the buffer size to 10, 20 or maybe even 100 for logging?

No, I don't think it is ever a good idea.

You want to know if an error occurs, isn't it? What happens if your app crashes with 99 error messages in the buffer? It will be hard to know what actually caused the problem...

If you have so many error messages, IMHO performance is the least of your problems. In a well working app, there should be very few errors, if at all. Most of the log messages should be of lower level (INFO / DEBUG / etc.) which can be filtered out with an appropriate level threshold in a live environment where performance is important.

Maybe using an appender that writes on MSMQ and a separate task to dequeue these messages and writing on DB allow you to ensure all message are logged without a performance decrease. Unfortunately an MSMQ appender does not come out of the box, but is easy to write one. In any case, since you are logging errors, are you really sure performance decrease by logging one by one directly? Usually applications does not fails so frequently that spending some times more during error reporting should not affect the overall performance. Try to avoid premature optimization, and have a test to see before write something more complex.

As with most things, it's a trade off: guaranteed capture of every logged error versus performance.

Only you know your business requirements. You should benchmark in your environment to determine an acceptable level of buffering.

As another poster has mentioned, a mixed approach would be best.

If the rate of errors is generally high, then waiting for 99 more errors to be logged isn't really going to be an issue for you unless the process comes crashing down. You'll have all of your errors in a timely fashion.

However, if the rate of errors is low, such that you'd be waiting a long time for even 10 errors to come through, then what is the point of batching? There is no point sitting around for minutes before finally dumping data to a database. A database can most certainly handle an insert every few seconds.. and probably a whole lot more than that.

Log4net offers what they call lossy logging. Basically you set up the appender to buffer for instance 100 messages (of all levels) and you tell the appender to flush the buffer as soon as an error is logged. In this mode the appender will discard log messages when the buffer is full, but depending on your requirements this may be perfectly acceptable (it is for me usually). The big advantage is that you get buffering and verbose logging without creating much overhead.

It would also be possible to instruct the appender to buffer without discarding messages. In that case it would flush the buffer if an error occurs or the buffer is full. So far this this was not a useful configuration for me.