Memory issues with InputStream in Java

I need to read a file into an array o开发者_StackOverflow社区f Bytes.The entire file needs to be read into the array. The problem is I am getting an OutOfMemory Error since the file size is too large. Increasing -XmX does not seem to have any effect. Here is the code snippet :

InputStream in = new FileInputStream(file);
long length = file.length();                        
byte[] out = new byte[(int)length]; 
// Process the byte array

The problem occurs during the byte array instantion. Is there a less memory intensive workaround to this issue?

You need to have far more free memory than the largest file you can have to use this approach. Given a machine with 24 GB costs less than £2K, this isn't as silly idea as it used to be. Actually the 2GB limit for a byte[] is more of a headache in some situations.

However, the usual way to read an InputStream is to read a block of say 8KB at a time. This way you only need to have far more than 8KB free.

BTW: You can use -mx1g instead of the option you are using.

No, if your file is too large to fit into memory, it's too large to fit in memory.

A better solution would be to try to process the stream as a stream, rather than loading the whole thing into memory. Without knowing what processing you're trying to achieve, we can't really tell whether or not that's feasible.

For example, if you're just trying to compute a secure hash of the file, you should be able to do that without loading significant amounts of data in at a time - but if your processing requires random access to data, you may need to use RandomAccessFile instead.

The workarround would be, not to load the entire file into the RAM. In fact you can't do this for large files, because you have to allocate lots of memory at one peace, which may not work.

The question ist: Do you really need the entire file in memory?

edit

InputStream in = new FileInputStream(file);
long length = file.length();     
// At this point a warning should appear, because the code would
// not work for files larger than Integer.MAX_VALUE                   
byte[] out = new byte[(int)length]; 

How about using a memory mapped file: FileChannel

From http://www.java-tips.org/java-se-tips/java.nio/how-to-create-a-memory-mapped-file-3.html:

try {
    File file = new File("filename");

    // Create a read-only memory-mapped file
    FileChannel roChannel = 
      new RandomAccessFile(file, "r").getChannel();

    ByteBuffer readonlybuffer = 
      roChannel.map(FileChannel.MapMode.READ_ONLY, 
0, (int)roChannel.size());

    // Create a read-write memory-mapped file
    FileChannel rwChannel = 
      new RandomAccessFile(file, "rw").getChannel();

    ByteBuffer writeonlybuffer= 
      rwChannel.map(FileChannel.MapMode.READ_WRITE, 
  0, (int)rwChannel.size());

    // Create a private (copy-on-write) memory-mapped file.
    // Any write to this channel results in a private 
    // copy of the data.
    FileChannel pvChannel = 
      new RandomAccessFile(file, "rw").getChannel();

    ByteBuffer privatebuffer = 
      roChannel.map(FileChannel.MapMode.READ_WRITE, 
  0, (int)rwChannel.size());

} catch (IOException e) {
}