Memory leak in JDBC4Connection

I'm trying to catch a memory leak in one of our Java daemons, and after dumping the memory and analyzing it via Memory Analyzer Tool, noticed that most of the leak is caused by the JDBC4Connection:

10 instances of "com.mysql.jdbc.JDBC4Connection", loaded by "sun.misc.Launcher$AppClassLoader @ 0x2aaab620ed00" occupy 858,283,752 (81.55%) bytes. Biggest instances:开发者_JS百科

* com.mysql.jdbc.JDBC4Connection @ 0x2aaab64ad820 - 87,110,160 (8.28%) bytes.
* com.mysql.jdbc.JDBC4Connection @ 0x2aaab64af520 - 86,730,408 (8.24%) bytes.
* com.mysql.jdbc.JDBC4Connection @ 0x2aaab64ad0e0 - 86,584,048 (8.23%) bytes.
* com.mysql.jdbc.JDBC4Connection @ 0x2aaab64aede0 - 86,488,800 (8.22%) bytes.
* com.mysql.jdbc.JDBC4Connection @ 0x2aaab61f5320 - 85,752,872 (8.15%) bytes.
* com.mysql.jdbc.JDBC4Connection @ 0x2aaab64ae6a0 - 85,603,280 (8.13%) bytes.
* com.mysql.jdbc.JDBC4Connection @ 0x2aaab64adf60 - 85,270,440 (8.10%) bytes.
* com.mysql.jdbc.JDBC4Connection @ 0x2aaab61f4be0 - 85,248,592 (8.10%) bytes.
* com.mysql.jdbc.JDBC4Connection @ 0x2aaab64afc60 - 85,120,704 (8.09%) bytes.
* com.mysql.jdbc.JDBC4Connection @ 0x2aaab61f5a60 - 84,374,448 (8.02%) bytes.

Keywords
com.mysql.jdbc.JDBC4Connection
sun.misc.Launcher$AppClassLoader @ 0x2aaab620ed00

I'm pretty sure that we closing all of the MySQL resources, but can't quite find what causing this.

Is there any good way to catch it? Have you experienced this in the past, and can advise what should I look for?

P.S.: Looking deeper with MAT, I see the following info:

com.mysql.jdbc.JDBC4Connection @ 0x2aaab64ad820 | 1,856 | 87,110,160 | 8.28% |- java.util.HashMap @ 0x2aaab62115a8 | 64 | 87,021,632 | 8.27% | '- java.util.HashMap$Entry[16384] @ 0x2aaae182e970| 131,096 | 87,021,568 | 8.27%

It seems as each JDBC holds a huge amount of Hashmap entries (>6000 objects), and does not free them at all.

Thanks in advance!

Duffymo is almost certainly right. In the past when we've had memory leaks, it's practically ALWAYS the MySQL JDBC driver. Just forgetting to close one little ResultSet or Connection or Statement somewhere. I ended up auditing the entire codebase for every time we used those to find the problem and ensure they get closed.

As for the HashMap, I've seen that too. I haven't looked at the source but my impression was that the MySQL driver stored the rows (at least row values) in HashMaps internally.

Leaking ResultSets is sadly easy. The idea of those closeable resources that take care of this themselves coming in JDK 7 or 8 really appeals to me for this reason.

You could insert a shim class somewhere (say for Connection) to log each opened/closed resource to see if you can catch where the leak is without directly reading all your source.

I'm pretty sure that we closing all of the MySQL resources

If not 100% sure, please show how you're closing your connections.

Are you using a connection pool? Would it happen to have a pool size of 10?

Sounds like you have a connection pool with 10 connections, and the pool has statement caching enabled, without having the cache size limited to a healthy amount.