Flood of EHCache threads spawned on startup

UPDATE: Still puzzled. I think the only possibility is an exception is thrown inside the init() method after the thread is spawned but the exception is not being logged. I catch all exceptions thrown in the create() method and log them to the warning log. Anyone have a theory about why Log4J wouldn't be able to print my log messages? The logging level is set to WARN.

I'm running a Spring 3.0 app using EHCache on JBoss. Sometimes when I start up the app server, the JVM completely shuts down within 20 or 30 seconds. I set it up to take a thread dump every 5 seconds and find that after about 15 seconds, the dump starts to get exponentially larger with this same thread:

"net.sf.ehcache.CacheManager@7d73124b" daemon prio=10 tid=0x00002aac1426a000 nid=0x2300 in Object.wait() [0x00002aac0d886000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00002aaaf5c556e0> (a java.util.TaskQueue)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x00002aaaf5c556e0> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

By the time the system shuts down there are about 10,000 threads that look exactly like this with just the memory addresses changed.

I did some research and found that EHCache's Cache Manager spawns this thread in the init() method, which is called by each of its constructors. The Cache Manager is supposed to be a singleton which is accessed through the create() method:

public static CacheManager create() throws CacheException {
    if (singleton != null) {
        return singleton;
    }
    synchronized (CacheManager.class) {
        if (singleton == null) {
            LOG.debug("Creating new CacheManager with default config");
            singleton = new CacheManager();
        } else {
            LOG.debug("Attempting to create an existing singleton. Existing singleton returned.");
        }
        return singleton;
    }
}

Having a thread dump every 5 seconds, I can see the stack traces as these threads multiply. There is always one thread inside that synchronize block whic开发者_StackOverflow社区h has made it to the constructor and will spawn a thread. Then there are always a few threads waiting to get into the synchronized block.

As far as I can tell, that null check on singleton inside the synchronized block should prevent more than one entry into the constructor, but somehow every thread appears to be finding singleton to be null. I log all exceptions to the warning log so I think if an exception were being thrown after the thread was created in the constructor, I would see it in the log. I don't see any warnings but I verified that that would cause this behavior.

Does anyone have any other idea how more than one thread could make it to that constructor? Would java cache the null check since it had just null checked the same field on the previous line?

Looking at the thread dump, it appears that the real problem might be a timer run amok, that continually spawns threads thataccess the cache manager. I think I'd first check your code, to verify that you're not firing a timer with a 0 wait time (could be the result of a bad time offset calculation).

If this are threads spawned by Cache Manager's init, then they're probably threads watching for version update at ehcache site (though i cannot be sure). Those checks can be suppressed either using Terracotta's runtime property (i don't remember its name) or cache configuration setting (some XML attribute or Spring setter, depending on how cache configuration is defined).

Attempting to create an existing singleton. Existing singleton returned.