Long lived Java WeakReferences

I am currently trying to diagnose a slow memory leak in my application. The facts I have so far are as follows.

• I have a heap dump from a 4 day run of the application.
• This heap dump contains ~800 WeakReference objects which point to objects (all of the same type, which I will call Foo for the purposes of this question) retaining 40mb of memory.
• Eclipse Memory Analysis Tool shows that each of the Foo objects referred to by these WeakReferences is not referred to by any other objects. My expectation is that this should make these Foo objects Weakly Reachable and thus they should be collected at the next GC.
• Each of these Foo objects has a timestamp which shows that they were allocated over the course of the 4 day run. I also have logs during this time which confirm that Garbage Collection was happening.
• A huge number of F开发者_如何学Pythonoo objects are being created by my application and only a very small fraction of them are ending up in this state within the heap dump. This suggests to me that the root cause is some sort of race condition.
• My application uses JNI to call through to a native library. The JNI code calls NewGlobalRef 4 times during start of day initialisation to get references to Java classes which it uses.

What could possibly cause these Foo classes to not be collected despite only being referenced by WeakReferences (according to Eclipse Memory Analyser Tool)?

EDIT1:

@mindas The WeakReference I am using is equivalent to the following example code.

public class FooWeakRef extends WeakReference<Foo>
{
  public long longA;
  public long longB;
  public String stringA;

  public FooWeakRef(Foo xiObject, ReferenceQueue<Foo> xiQueue)
  {
    super(xiObject, xiQueue);
  }
}

Foo does not have a finalizer and any finalizer would not be a consideration so long as the WeakRefs have not been cleared. An object is not finalizable when it is weakly reachable. See this page for details.

@kasten The weakreferences are cleared before the object is finalizable. My heap dump shows that this has not happened.

@jarnbjo I refer to the WeakReference Javadoc:

"Suppose that the garbage collector determines at a certain point in time that an object is weakly reachable. At that time it will atomically clear all weak references to that object and all weak references to any other weakly-reachable objects from which that object is reachable through a chain of strong and soft references."

This suggests to me that the GC should be detecting the fact that my Foo objects are "Weakly reachable" and "At that time" clearing the weak references.

EDIT 2

@j flemm - I know that 40mb doesn't sound like much but I am worried that 40mb in 4 days means 4000mb in 100 days. All of the docs I have read suggest that objects which are weakly reachable should not hang around for several days. I am therefore interested in any other explanations about how an object could be strongly referenced without the reference showing up in a heap dump.

I am going to try allocating some large objects when some of these dangling Foo objects are present and see whether the JVM collects them. However, this test will take a couple of days to setup and complete.

EDIT 3

@jarnbjo - I understand that I have no guarantee about when the JDK will notice that an object is weakly reachable. However, I would expect that an application under heavy load for 4 days would provide enough opportunities for the GC to notice that my objects are weakly reachable. After 4 days I am strongly suspicious that the remaining weakly references objects have been leaked somehow.

EDIT 4

@j flemm - Thats really interesting! Just to clarify, are you saying that GC is happening on your app and is not clearing Soft/Weak refs? Can you give me any more details about what JVM + GC Config you are using? My app is using a memory bar at 80% of the heap to trigger GC. I was assuming that any GC of the old gen would clear Weak refs. Are you suggesting that a GC only collects Weak refs once the memory usage is above a higher threshold? Is this higher limit configurable?

EDIT 5

@j flemm - Your comment about clearing out WeakRefs before SoftRefs is consistent with the Javadoc which states: SoftRef: "Suppose that the garbage collector determines at a certain point in time that an object is softly reachable. At that time it may choose to clear atomically all soft references to that object and all soft references to any other softly-reachable objects from which that object is reachable through a chain of strong references. At the same time or at some later time it will enqueue those newly-cleared soft references that are registered with reference queues."

WeakRef: "Suppose that the garbage collector determines at a certain point in time that an object is weakly reachable. At that time it will atomically clear all weak references to that object and all weak references to any other weakly-reachable objects from which that object is reachable through a chain of strong and soft references. At the same time it will declare all of the formerly weakly-reachable objects to be finalizable. At the same time or at some later time it will enqueue those newly-cleared weak references that are registered with reference queues."

For clarity, are you saying that the Garbage Collector runs when your app has more than 50% free memory and in this case it does not clear WeakRefs? Why would the GC run at all when your app has >50% free memory? I think your app is probably just generating a very low amount of garbage and when the collector runs it is clearing WeakRefs but not SoftRefs.

EDIT 6

@j flemm - The other possible explanation for your app's behaviour is that the young gen is being collected but that your Weak and Soft refs are all in the old gen and are only cleared when the old gen is being collected. For my app I have stats showing that the old gen is being collected which should mean that WeakRefs get cleared.

EDIT 7

I am starting a bounty on this question. I am looking for any plausible explanations for how WeakRefs could fail to be cleared while GC is happening. If the answer is that this is impossible I would ideally like to be pointed at the appropriate bits of OpenJDK which show WeakRefs being cleared as soon as an object is determined to be weakly reachable and that weak reachability is resolved every time GC runs.

I have finally got round to checking the Hotspot JVM source code and found the following code.

In referenceProcessor.cpp:

void ReferenceProcessor::process_discovered_references(
  BoolObjectClosure*           is_alive,
  OopClosure*                  keep_alive,
  VoidClosure*                 complete_gc,
  AbstractRefProcTaskExecutor* task_executor) {
  NOT_PRODUCT(verify_ok_to_handle_reflists());

  assert(!enqueuing_is_done(), "If here enqueuing should not be complete");
  // Stop treating discovered references specially.
  disable_discovery();

  bool trace_time = PrintGCDetails && PrintReferenceGC;
  // Soft references
  {
    TraceTime tt("SoftReference", trace_time, false, gclog_or_tty);
    process_discovered_reflist(_discoveredSoftRefs, _current_soft_ref_policy, true,
                               is_alive, keep_alive, complete_gc, task_executor);
  }

  update_soft_ref_master_clock();

  // Weak references
  {
    TraceTime tt("WeakReference", trace_time, false, gclog_or_tty);
    process_discovered_reflist(_discoveredWeakRefs, NULL, true,
                               is_alive, keep_alive, complete_gc, task_executor);
  }

The function process_discovered_reflist has the following signature:

void
ReferenceProcessor::process_discovered_reflist(
  DiscoveredList               refs_lists[],
  ReferencePolicy*             policy,
  bool                         clear_referent,
  BoolObjectClosure*           is_alive,
  OopClosure*                  keep_alive,
  VoidClosure*                 complete_gc,
  AbstractRefProcTaskExecutor* task_executor)

This shows that WeakRefs are being unconditionally cleared by ReferenceProcessor::process_discovered_references.

Searching the Hotspot code for process_discovered_reference shows that the CMS collector (which is what I am using) calls this method from the following call stack.

CMSCollector::refProcessingWork
CMSCollector::checkpointRootsFinalWork
CMSCollector::checkpointRootsFinal

This call stack looks like it is invoked every time a CMS collection is run.

Assuming this is true, the only explanation for a long lived weakly referenced object would be either a subtle JVM bug or if the GC had not been run.

You might want to check if you have leaked classloader issue. More on this topic you could find in this blog post

You need to clarify on what is the link between Foo and WeakReference. The case

class Wrapper<T> extends WeakReference<T> {
  private final T referent;
  public Wrapper(T referent) {
    super(t);
    this.referent = referent;
  }
}

is very different from just

class Wrapper<T> extends WeakReferece<T> {
  public Wrapper(T referent) {
    super(t);
  }
}

or its inlined version, WeakReference<Foo> wr = new WeakReference<Foo>(foo).

So I assume your case is not like I described in my first code snippet.

As you have said you are working with JNI, you might want to check if you have any unsafe finalizers. Every finalizer should have finally block calling super.finalize() and it's easy to slip.

You probably need to tell us more about the nature of your objects to offer better ideas.

Try SoftReference instead. Javadoc says: All soft references to softly-reachable objects are guaranteed to have been cleared before the virtual machine throws an OutOfMemoryError.

WeakReference doesn't have such guarantees, which makes them more suitable for caches, but sometimes SoftReferences are better.

@iirekm No: WeakReferences are 'weaker' than SoftReferences, meaning that a WeakReference will always be garbage collected before a SoftReference.

More info in this post: Understanding Java's Reference classes: SoftReference, WeakReference, and PhantomReference

Edit: (after reading comments) Yes surely Weak References are 'Weaker' than SoftReferences, typo. :S

Here's some use cases to throw further light on the subject:

• SoftReference: In-memory cache (Object stays alive until VM deems that there's not enough heap mem)
• WeakReference: Auto-clearing Listeners (Object should be cleared on next GC cycle after deemed being Weakly reachable)
• PhantomReference: Avoiding out-of-memory errors when handling unusually large objects (When scheduled in reference queue, we know that host object is to be cleared, safe to allocate another large object). Think of it as a finalize() alternative, without the ability to bring dead objects back to life (as you potentially could with finalize)

This being said, nothing prevents the VM (please correct me if I'm wrong) to let the Weakly reachable objects stay alive as long as it is not running out of memory (as in the orig. author's case).

This is the best resource I could find on the subject: http://www.pawlan.com/monica/articles/refobjs/

Edit 2: Added "to be" in front of cleared in PhantomRef

I am not acquainted with Java, but you may be using a generational garbage collector, which will keep your Foo and FooWeakRef objects alone (not collected) as long as

• they passed in an older generation
• there is enough memory to allocate new objects in younger generations

Does the log that indicates that garbage collection occurred discriminates between major and minor collections?

For non-believers who claim that weak references are cleared before soft references:

import java.lang.ref.Reference;
import java.lang.ref.ReferenceQueue;
import java.lang.ref.SoftReference;
import java.lang.ref.WeakReference;
import java.util.HashMap;
import java.util.Map;


public class Test {

/**
 * @param args
 */
public static void main(String[] args) {
    ReferenceQueue<Object> q = new ReferenceQueue<Object>();
    Map<Reference<?>, String> referenceToId = new HashMap<Reference<?>, String>();
    for(int i=0; i<100; ++i) {
        Object obj = new byte [10*1024*1024];    // 10M
        SoftReference<Object> sr = new SoftReference<Object>(obj, q);
        referenceToId.put(sr, "soft:"+i);
        WeakReference<Object> wr = new WeakReference<Object>(obj, q);
        referenceToId.put(wr, "weak:"+i);

        for(;;){
            Reference<?> ref = q.poll();
            if(ref == null) {
                break;
            }
            System.out.println("cleared reference " + referenceToId.get(ref) + ", value=" + ref.get());
        }
    }
}
}

If your run it with either -client or -server, you'll see that soft references are always cleared before weak references, which also agrees with Javadoc: http://download.oracle.com/javase/1.4.2/docs/api/java/lang/ref/package-summary.html#reachability

Typically soft/weak references are used in connection with Maps to make kinds of caches. If keys in your Map are compared with == operator, (or unoverriden .equals from Object), then it's best to use Map which operates on SoftReference keys (eg from Apache Commons) - when the object 'disappears' no other object will ever be equal in the '==' sense to the old one. If keys of your Map are compared with advanced .equals() operator, like String or Date, many other objects may match to the 'disappearing' one, so it's better to use the standard WeakHashMap.