Java using more memory than the allocated memory

An Apache Tomcat (Atlassian Confluence) instance is started using the following Java options:

JAVA_OPTS="-Xms256m -Xmx512m -XX:MaxPermSize=256m -Djava.awt.headless=true "

However I see that after starting up it quickly eats through most of the 1GB of memory that is available on my virtual server.

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 6082 root      19   0 1105m 760m  16m S  0.7 74.2   5:20.51 java

Shouldn't the overall consumed memory (heap + PermGen) stay under what is specified using -Xmx? One of the problems this is causing is that I cannot shutdown the server using the shutdown script since it tries to spawn a JVM with 256MB of memory which fails because of it not being available.

For example, a native library can easily allocate memory outside Java heap.

Direct ByteBuffer also does that: http://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html

The contents of direct buffers may reside outside of the normal garbage-collected heap, and so their impact upon the memory footprint of an application might not be obvious.

There are good reasons to allocate huge direct ByteBuffers.

http://ehcache.org/documentation/offheap_store.html

Total Tomcat memory consumption should be calculated at NO LESS THAN Xmx + XX:MaxPermSize (in your case, 768MB), but I do recall seeing somewhere that it can go over that. Xmx is only the heap space, and PermGen is outside the heap (kind of).