Why does a BufferedImage require so much memory beyond the size of its data array?

I'm trying to determine how much heap any given TYPE_INT_ARGB BufferedImage will use so that, for a program which is doing some image processing, I can set a reasonable max heap based on the size of image we feed it.

I wrote the following program as a test, which I then used to determine the least maximum heap under which it would run without an OutOfMemoryError:

import java.awt.image.BufferedImage;

public class Test {
  public static void main(String[] args) {
    final int w = Integer.parseInt(args[0]);
    final int h = Integer.parseInt(args[1]);

    final BufferedImage img =
      new BufferedImage(w, h, BufferedImage.TYPE_INT_ARGB);

    System.out.println((4*w*h) >> 20);
  }
}

(The printed value is the expected size of the int[] in which the BufferedImage's pixel data is stored.) What I expected to find was that the required max heap is something like x + c, wh开发者_JAVA百科ere x is the size of the data array and c is a constant consisting of the sizes of the classes which are loaded, the BufferedImage object, etc. This is what I found instead (all values are in MB):

4*w*h   min max heap
-----   ------------
  5          -
 10         15
 20         31
 40         61
 80        121
160        241

1.5x is a good fit for the observations. (Note that I found no minimum for the 5MB image.) I don't understand what I'm seeing. What are these extra bytes?

There seem to be a bug in Oracle's VM introduced somewhere between 1.6.0_16 and 1.6.0_20. You can even reduce the problem to allocating an int array, as the problem is not only related to BufferedImage.

With 1.6.0_16, I need at least 413 MB heap to allocate an int array with 100,000,000 elements, which seem reasonable. With 1.6.0_20, the same operation requires at least 573 MB heap space, although only appr 400,000,000 bytes are actually used after allocating the array.

On further investigation, the problem appears to be that the Old Generation in the heap is not able to expand sufficiently to accommodate the image's data array, despite that there is enough free memory in the heap at large.

For further details about how to expand the Old Generation, see this question.

The problem is that the BufferedImage object stores the image in the memory in uncompresed format. There is an effective solution for this: you can store the image on the hard drive and you don't have to worry about the heap size or the physical memory limit. It can store maximum 2,147,483,647 pixels (or 46,340 x 46,340 pixels). BigBufferedImage solves this problem.

Create az empty a BigBufferedImage:

BigBufferedImage image = BigBufferedImage.create(
        tempDir, width, height, type);

Load an existent image to BigBufferedImage:

BigBufferedImage image = BigBufferedImage.create(
        imagePath, tempDir, type);

Render the part of the image:

part = image.getSubimage(x, y, width, height);

The implementation of BigBufferedImage:

package com.pulispace.mc.ui.panorama.util;

/*
 * This class is part of MCFS (Mission Control - Flight Software) a development 
 * of Team Puli Space, official Google Lunar XPRIZE contestant. 
 * This class is released under Creative Commons CC0.
 * @author Zsolt Pocze, Dimitry Polivaev
 * Please like us on facebook, and/or join our Small Step Club.
 * http://www.pulispace.com
 * https://www.facebook.com/pulispace
 * http://nyomdmegteis.hu/en/
 */
import java.awt.Point;
import java.awt.Rectangle;
import java.awt.color.ColorSpace;
import java.awt.image.BandedSampleModel;
import java.awt.image.BufferedImage;
import java.awt.image.ColorModel;
import java.awt.image.ComponentColorModel;
import java.awt.image.DataBuffer;
import java.awt.image.Raster;
import java.awt.image.RenderedImage;
import java.awt.image.SampleModel;
import java.awt.image.WritableRaster;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.Hashtable;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.imageio.ImageIO;
import javax.imageio.ImageReadParam;
import javax.imageio.ImageReader;
import javax.imageio.stream.ImageInputStream;
import sun.nio.ch.DirectBuffer;

public class BigBufferedImage extends BufferedImage {

    private static final String TMP_DIR = System.getProperty("java.io.tmpdir");
    public static final int MAX_PIXELS_IN_MEMORY =  1024 * 1024;

    public static BufferedImage create(int width, int height, int imageType) {
        if (width * height > MAX_PIXELS_IN_MEMORY) {
            try {
                final File tempDir = new File(TMP_DIR);
                return createBigBufferedImage(tempDir, width, height, imageType);
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        } else {
            return new BufferedImage(width, height, imageType);
        }
    }

    public static BufferedImage create(File inputFile, int imageType) throws IOException {
        try (ImageInputStream stream = ImageIO.createImageInputStream(inputFile);) {
            Iterator<ImageReader> readers = ImageIO.getImageReaders(stream);
            if (readers.hasNext()) {
                try {
                    ImageReader reader = readers.next();
                    reader.setInput(stream, true, true);
                    int width = reader.getWidth(reader.getMinIndex());
                    int height = reader.getHeight(reader.getMinIndex());
                    BufferedImage image = create(width, height, imageType);
                    int cores = Math.max(1, Runtime.getRuntime().availableProcessors() / 2);
                    int block = Math.min(MAX_PIXELS_IN_MEMORY / cores / width, (int) (Math.ceil(height / (double) cores)));
                    ExecutorService generalExecutor = Executors.newFixedThreadPool(cores);
                    List<Callable<ImagePartLoader>> partLoaders = new ArrayList<>();
                    for (int y = 0; y < height; y += block) {
                        partLoaders.add(new ImagePartLoader(
                            y, width, Math.min(block, height - y), inputFile, image));
                    }
                    generalExecutor.invokeAll(partLoaders);
                    generalExecutor.shutdown();
                    return image;
                } catch (InterruptedException ex) {
                    Logger.getLogger(BigBufferedImage.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
        }
        return null;
    }

    private static BufferedImage createBigBufferedImage(File tempDir, int width, int height, int imageType)
        throws FileNotFoundException, IOException {
        FileDataBuffer buffer = new FileDataBuffer(tempDir, width * height, 4);
        ColorModel colorModel = null;
        BandedSampleModel sampleModel = null;
        switch (imageType) {
            case TYPE_INT_RGB:
                colorModel = new ComponentColorModel(ColorSpace.getInstance(ColorSpace.CS_sRGB),
                    new int[]{8, 8, 8, 0},
                    false,
                    false,
                    ComponentColorModel.TRANSLUCENT,
                    DataBuffer.TYPE_BYTE);
                sampleModel = new BandedSampleModel(DataBuffer.TYPE_BYTE, width, height, 3);
                break;
            case TYPE_INT_ARGB:
                colorModel = new ComponentColorModel(ColorSpace.getInstance(ColorSpace.CS_sRGB),
                    new int[]{8, 8, 8, 8},
                    true,
                    false,
                    ComponentColorModel.TRANSLUCENT,
                    DataBuffer.TYPE_BYTE);
                sampleModel = new BandedSampleModel(DataBuffer.TYPE_BYTE, width, height, 4);
                break;
            default:
                throw new IllegalArgumentException("Unsupported image type: " + imageType);
        }
        SimpleRaster raster = new SimpleRaster(sampleModel, buffer, new Point(0, 0));
        BigBufferedImage image = new BigBufferedImage(colorModel, raster, colorModel.isAlphaPremultiplied(), null);
        return image;
    }

    private static class ImagePartLoader implements Callable<ImagePartLoader> {

        private final int y;
        private final BufferedImage image;
        private final Rectangle region;
        private final File file;

        public ImagePartLoader(int y, int width, int height, File file, BufferedImage image) {
            this.y = y;
            this.image = image;
            this.file = file;
            region = new Rectangle(0, y, width, height);
        }

        @Override
        public ImagePartLoader call() throws Exception {
            Thread.currentThread().setPriority((Thread.MIN_PRIORITY + Thread.NORM_PRIORITY) / 2);
            try (ImageInputStream stream = ImageIO.createImageInputStream(file);) {
                Iterator<ImageReader> readers = ImageIO.getImageReaders(stream);
                if (readers.hasNext()) {
                    ImageReader reader = readers.next();
                    reader.setInput(stream, true, true);
                    ImageReadParam param = reader.getDefaultReadParam();
                    param.setSourceRegion(region);
                    BufferedImage part = reader.read(0, param);
                    Raster source = part.getRaster();
                    WritableRaster target = image.getRaster();
                    target.setRect(0, y, source);
                }
            }
            return ImagePartLoader.this;
        }
    }

    private BigBufferedImage(ColorModel cm, SimpleRaster raster, boolean isRasterPremultiplied, Hashtable<?, ?> properties) {
        super(cm, raster, isRasterPremultiplied, properties);
    }

    public void dispose() {
        ((SimpleRaster) getRaster()).dispose();
    }

    public static void dispose(RenderedImage image) {
        if (image instanceof BigBufferedImage) {
            ((BigBufferedImage) image).dispose();
        }
    }

    private static class SimpleRaster extends WritableRaster {

        public SimpleRaster(SampleModel sampleModel, FileDataBuffer dataBuffer, Point origin) {
            super(sampleModel, dataBuffer, origin);
        }

        public void dispose() {
            ((FileDataBuffer) getDataBuffer()).dispose();
        }

    }

    private static final class FileDataBufferDeleterHook extends Thread {

        static {
            Runtime.getRuntime().addShutdownHook(new FileDataBufferDeleterHook());
        }

        private static final HashSet<FileDataBuffer> undisposedBuffers = new HashSet<>();

        @Override
        public void run() {
            final FileDataBuffer[] buffers = undisposedBuffers.toArray(new FileDataBuffer[0]);
            for (FileDataBuffer b : buffers) {
                b.disposeNow();
            }
        }
    }

    private static class FileDataBuffer extends DataBuffer {

        private final String id = "buffer-" + System.currentTimeMillis() + "-" + ((int) (Math.random() * 1000));
        private File dir;
        private String path;
        private File[] files;
        private RandomAccessFile[] accessFiles;
        private MappedByteBuffer[] buffer;

        public FileDataBuffer(File dir, int size) throws FileNotFoundException, IOException {
            super(TYPE_BYTE, size);
            this.dir = dir;
            init();
        }

        public FileDataBuffer(File dir, int size, int numBanks) throws FileNotFoundException, IOException {
            super(TYPE_BYTE, size, numBanks);
            this.dir = dir;
            init();
        }

        private void init() throws FileNotFoundException, IOException {
            FileDataBufferDeleterHook.undisposedBuffers.add(this);
            if (dir == null) {
                dir = new File(".");
            }
            if (!dir.exists()) {
                throw new RuntimeException("FileDataBuffer constructor parameter dir does not exist: " + dir);
            }
            if (!dir.isDirectory()) {
                throw new RuntimeException("FileDataBuffer constructor parameter dir is not a directory: " + dir);
            }
            path = dir.getPath() + "/" + id;
            File subDir = new File(path);
            subDir.mkdir();
            buffer = new MappedByteBuffer[banks];
            accessFiles = new RandomAccessFile[banks];
            files = new File[banks];
            for (int i = 0; i < banks; i++) {
                File file = files[i] = new File(path + "/bank" + i + ".dat");
                final RandomAccessFile randomAccessFile = accessFiles[i] = new RandomAccessFile(file, "rw");
                buffer[i] = randomAccessFile.getChannel().map(FileChannel.MapMode.READ_WRITE, 0, getSize());
            }
        }

        @Override
        public int getElem(int bank, int i) {
            return buffer[bank].get(i) & 0xff;
        }

        @Override
        public void setElem(int bank, int i, int val) {
            buffer[bank].put(i, (byte) val);
        }

        @Override
        protected void finalize() throws Throwable {
            dispose();
        }

        private void disposeNow() {
            final MappedByteBuffer[] disposedBuffer = this.buffer;
            this.buffer = null;
            disposeNow(disposedBuffer);
        }

        public void dispose() {
            final MappedByteBuffer[] disposedBuffer = this.buffer;
            this.buffer = null;
            new Thread() {
                @Override
                public void run() {
                    disposeNow(disposedBuffer);
                }
            }.start();
        }

        private void disposeNow(final MappedByteBuffer[] disposedBuffer) {
            FileDataBufferDeleterHook.undisposedBuffers.remove(this);
            if (disposedBuffer != null) {
                for (MappedByteBuffer b : disposedBuffer) {
                    ((DirectBuffer) b).cleaner().clean();
                }
            }
            if (accessFiles != null) {
                for (RandomAccessFile file : accessFiles) {
                    try {
                        file.close();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                }
                accessFiles = null;
            }
            if (files != null) {
                for (File file : files) {
                    file.delete();
                }
                files = null;
            }
            if (path != null) {
                new File(path).delete();
                path = null;
            }
        }

    }
}