Dealing with huge (potentially over 30000x30000) images in Python?

I'm tryi开发者_StackOverflowng to use a python script called deepzoom.py to convert large overhead renders (often over 1GP) to the Deep Zoom image format (ie, google maps-esque tile format), but unfortunately it's powered by PIL, which usually ends up crashing due to memory limitations. The creator has said he's delving into VIPS, but even nip2 (the GUI frontend for VIPS) fails to open the image. In another question by someone else (though on the same topic), someone suggested OpenImageIO, which looks like it has the ability, and has Python wrappers, but there aren't any proper binaries provided, and trying to compile it on Windows is a nightmare.

Are there any alternative libraries for Python I can use? I've tried PythonMagickWand (wrapper for ImageMagick) and PythonMagick (wrapper for GraphicsMagick), but both of those also run into memory problems.

I had a very similar problem and I ended up solving it by using netpbm, which works fine on windows. Netpbm had no problem with converting huge .png files and then slicing, cropping, re-combining (using pamcrop, pamdice, and pamundice) and converting back to .png without using much memory at all. I just included the necessary netpbm binaries and dlls with my application and called them from python.

It sounds like you're trying to use georeferenced imagery or something similar, for which a GIS solution sounds more appropriate. I'd use GDAL -- it's an excellent library and comes with easy-to-use Python bindings via Swig.

On Windows, the easiest way to install it is via Frank Warmerdam's FWTools package.

I'm able to use pyvips to read images with size (50000, 50000, 3):

img = pyvips.Image.new_from_file('xxx.jpg')
arr = np.ndarray(buffer=img.write_to_memory(),
                  dtype=np.uint8,
                  shape=[img.height, img.width, img.bands])

Is a partial load useful? If you use PIL and the image format is .BMP: you can open() an image file (which doesn't load it), then do a crop(), and then load - which will only actually load the part of the image which you've selected by crop. Will probably also work with TGA, maybe even for JPG and less efficiently for PNG and other formats.

libvips comes with a very fast DeepZoom creator that can work with images of any size. Try:

$ vips dzsave huge.tif mydz

Will write the tiles to mydz_files and also write a mydz.dzi info file for you. It's typically 10x faster than deepzoom.py and has no size limit.

See this chapter in the manual for an introduction to dzsave.

You can do the same thing from Python using pyvips like this:

import pyvips

my_image = pyvips.Image.new_from_file("huge.tif", access="sequential")
my_image.dzsave("mydz")

The access="sequential" tells pyvips it can stream the image rather than having to read the whole thing into memory.