Huge memory footprint making native C calls from Java using JNA
I have a native C library that runs some algorithms on very large datasets (on the order of hundreds of mb to gb). This is being called from within a Java framework using JNA. Java loads the data and passes it through JNA to the C library.
The problem is, there seems to be an inordinate amount of memory being used. For one dataset, the process uses about 3.0gb after all the loading is finished on the Java end, and 2.0gb is in use by the C library (as determined using in-house memory management). But the process, once the C library is called, ends up maxing out at about 9.5gb!
Specific questions, then:
Is there no overlap between the Java and C side? That is to say, does JNA produce a C-valid copy of the Java data (all int and double arrays, by the way) and pass that to the native library instead of the same blocks that contain the data in Java?
Even presuming there is no overlap, and the native library is using a copy of the data contained within the JVM, where is that extra 4.5gb coming from? That about doubles the amount of system memory taken up by the process, and i can't imagine where it's all 开发者_如何学Pythongoing. The documentation on these aspects of JNA seems very limited, but I wonder if someone more familiar than I am with JNA might know why it's using so much memory, and if and how I might be able to reduce its footprint.
EDIT: The JNA-enabled Java class looks like this:
public interface MyNativeLibrary extends Library {
MyNativeLibrary INSTANCE = (MyNativeLibrary) Native.loadLibrary(
"native_library", MyNativeLibrary.class);
int native_library_function(int num_inputs, int inputs[], int max_num_outputs, int preallocated_outputs[]);
}
In this case, the return value of the native function would be the number of outputs returned, or an error code. The C interface is specified with int32_t, to make sure the sizes match.
If you want to allocate memory only on the native side, use the Memory class in JNA and access the memory using java.nio.ByteBuffer (Memory has a getByteBuffer() method). You can pass your arrays of integers as JNA Pointers the allocated memory. This should save you quite a bit of space and improve performance. Just make sure you manage the native memory resources on the Java side.
精彩评论