Log4j deadlock occuring between Logger and Appender

I am using a LogAppender class which extends a Console Appender and internally uses a Sendmail function to send mails if the log level is ERROR. (I know I could have used a SMPT appender for mailing, but this is how the infrastructure is so any please avoid any comments on this).

My application is split into different threads, and the issue mentioned below happends sporadically.(But some observation tells me that it might be due to one thread taking more time to initalize).

I get the following dump in jstack:

    "pool-68-thread-1":
  waiting to lock Monitor@0x0000000005496818 (Object@0xfffffd7e390328b0, a crazycode/api/core/common/log/LogAppender),
  which is held by "main"
"main":
  waiting to lock Monitor@0x00000000064dbfc0 (Object@0xfffffd7dfb1f8908, a org/apache/log4j/Logger),
  which is held by "pool-68-thread-1"

Found a total of 1 deadlock.

Thread t@328: (state = BLOCKED)
 - org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) @bci=13, line=202 (Interpreted frame)
 - org开发者_如何学Go.apache.log4j.Category.forcedLog(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=14, line=388 (Interpreted frame)
 - org.apache.log4j.Category.log(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=34, line=853 (Compiled frame)
 - crazycode.team2Api.common.logging.DESLogger.debug(java.lang.Object) @bci=12, line=240 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.refreshCache(java.util.List, java.lang.Object, java.lang.String) @bci=33, line=229 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.initialize(java.lang.Object) @bci=70, line=285 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.init() @bci=18, line=110 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.initialize() @bci=1, line=98 (Interpreted frame)
 - crazycode.api.alert.handler.MemoHandler.initRealHandler() @bci=4, line=573 (Interpreted frame)
 - crazycode.api.alert.handler.RealAlertCoordinator.initHandler(crazycode.api.alert.handler.AlertHandler, int, int, boolean, boolean) @bci=61, line=47 (Interpreted frame)
 - crazycode.api.alert.handler.run.BaseAlertHandlerRun$1$1.run() @bci=72, line=229 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=441 (Interpreted frame)
 - java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=4, line=138 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread t@327: (state = BLOCKED)
 - org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) @bci=13, line=202 (Interpreted frame)
 - org.apache.log4j.Category.forcedLog(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=14, line=388 (Interpreted frame)
 - org.apache.log4j.Category.log(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=34, line=853 (Compiled frame)
 - crazycode.team2Api.common.logging.DESLogger.debug(java.lang.Object) @bci=12, line=240 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.refreshCache(java.util.List, java.lang.Object, java.lang.String) @bci=33, line=229 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.initialize(java.lang.Object) @bci=70, line=285 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.init() @bci=18, line=110 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.initialize() @bci=1, line=98 (Interpreted frame)
 - crazycode.api.alert.handler.ResearchDocumentHandler.initRealHandler() @bci=86, line=957 (Interpreted frame)
 - crazycode.api.alert.handler.RealAlertCoordinator.initHandler(crazycode.api.alert.handler.AlertHandler, int, int, boolean, boolean) @bci=61, line=47 (Interpreted frame)
 - crazycode.api.alert.handler.run.BaseAlertHandlerRun$1$1.run() @bci=72, line=229 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=441 (Interpreted frame)
 - java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=4, line=138 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread t@322: (state = BLOCKED)
 - org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) @bci=13, line=202 (Interpreted frame)
 - org.apache.log4j.Category.forcedLog(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=14, line=388 (Interpreted frame)
 - org.apache.log4j.Category.log(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=34, line=853 (Compiled frame)
 - crazycode.team2Api.common.logging.DESLogger.debug(java.lang.Object) @bci=12, line=240 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.refreshCache(java.util.List, java.lang.Object, java.lang.String) @bci=33, line=229 (Interpreted frame)
 - crazycode.api.core.cache.management.CacheFiller.initialize(java.lang.Object) @bci=70, line=285 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.init() @bci=18, line=110 (Interpreted frame)
 - crazycode.api.alert.impl.service.ResearchAlertConfigCacheImpl.initialize() @bci=1, line=98 (Interpreted frame)
 - crazycode.api.alert.handler.RkdResearchDocumentHandler.initRealHandler() @bci=86, line=779 (Interpreted frame)
 - crazycode.api.alert.handler.RealAlertCoordinator.initHandler(crazycode.api.alert.handler.AlertHandler, int, int, boolean, boolean) @bci=61, line=47 (Interpreted frame)
 - crazycode.api.alert.handler.run.BaseAlertHandlerRun$1$1.run() @bci=72, line=229 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=441 (Interpreted frame)
 - java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=4, line=138 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)

Can someone shed some light as to what could be the issue and what are the ways to solve it??

I saw a similar trace today in my own code. Maybe it applies to you too?

Summary:

• Before (deadlock): logger.info(x); (where x is a complicated object that acquires a read lock within toString())
• After (ok): logger.info(x.toString());

Details:

In one thread I called logger.info(x), which called Category.callAppenders(LoggingEvent), which synchronizes on an appender and then calls x.toString(), which tried to acquire a read lock on x and blocked.

In another thread, I had a method that held the write lock on x while calling logger.info(""), which also called callAppenders, tried to synchronize on an appender, and blocked. Here's the relevant part of the trace:

Thread One [WAITING]
sun.misc.Unsafe.park(native method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
(... my ReadWriteLock$ReadLock.lock(Lock.java:256))
(... my Object's toString() ...)
org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
org.apache.log4j.xml.XMLLayout.format(XMLLayout.java:148)
org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
org.apache.log4j.Category.callAppenders(Category.java:206)
org.apache.log4j.Category.forcedLog(Category.java:391)
org.apache.log4j.Category.log(Category.java:856)
Log.ndebug(Log.java:965)



Thread Two [BLOCKED; waiting to lock org.apache.log4j.Logger@1623f]
org.apache.log4j.Category.callAppenders(Category.java:204)
org.apache.log4j.Category.forcedLog(Category.java:391)
org.apache.log4j.Category.log(Category.java:856)
Log.debug(Log.java:820)
(.. something that acquires my reentrant ReadWriteLock$WriteLock ...)

Are there stacks for the "main" thread and "pool-68-thread-1" threads? I'd look at the first line down the stack that's your code in each of those threads to figure out what's possibly causing the deadlock.