Large Image Processing with ImageMagick convert - Need More Throughput

I am converting some largish images from a multi-image (pyramidal) tif to png format. The salient parts of the report from "identity -verbose" on the largest image are here:

Image: 
  Format: TIFF (Tagged Image File Format)
  Class: DirectClass
  Geometry: 72224x64080+0+0
  Resolution: 72x72
  Print size: 1003.11x890
  Units: PixelsPerInch
  Type: TrueColor
  Base type: TrueColor
  Endianess: MSB
  Colorspace: RGB
  Depth: 8-bit
  Channel depth:
    red: 8-bit
    green: 8-bit
    blue: 8-bit
    ...
  Page geometry: 72224x64080+0+0
    ...
  Scene: 2 of 12
  Compression: JPEG
  Orientation: TopLeft
  Properties:
    ...
  Filesize: 1.389GBB
  Number pixels: 4.6281GB
  Pixels per second: 5.516MB
  User time: 218.277u
  Elapsed time: 13:60.020
  Version: ImageMagick 6.7.1-0 2011-07-06 Q16 http://www.imagemagick.org

I am intending to use deepzoom composer to produce input for the Silverlight multiscaleimage control with this image. My question is how do I bring my system to its knees while processing these images with ImageMagick - it is taking too long to convert them. I have looked at a few articles, but I can't seem to get anywhere.

Some system and other related information:

 OS: Windows 7 64 bit.
 CPU: Intel Core2 Duo E7300 @ 2.66, 2.67
 RAM: 4.0 GB
 PAGEFILE: 8-12GB on non-OS disk
 "MAGICK_TMPDIR": Yet another empty, non-os disk with 140GB available.

Here is the result of "identify -list resource":

 File         Area       Memory          Map         Disk    Thread
 ------------------------------------------------------------------
 1536     4.1582GB    15.491GiB    30.981GiB    unlimited         2

I am running this command to extract the image referenced above:

convert "myFN.tif[2]" -limit file 8192GB -limit thread 32 "myFN%d.png"

Adding the two limit values did not seem to make a difference. When I run this, I average about 10% CPU utilization and have a pagefile commit size of 3BG. I can barely tell that it is runnin开发者_开发技巧g.

Q1) Is there anything else I can do to get ImageMagick to use more system resources? Most of the "large image" links I have found are asking the opposite question.

Q2) Changing "policy.xml" values (such as files) located here: C:\Program Files\ImageMagick-6.7.1-Q16\www\source did not seem to affect anything - the changes did not show up in the next "identify -list resource." Is there a trick to this?

Q3) Any other hints or ideas for this task?

Thanks, David

libvips can convert pyramidal tiff directly into deepzoom pyramids. It's free, very fast and doesn't need much memory.

For example, I see:

$ vipsheader vips-pyr.tif 
vips-pyr.tif: 18008x22764 uchar, 3 bands, srgb, tiffload
$ time vips dzsave vips-pyr.tif x.zip
real    0m9.763s
user    0m19.700s
sys 0m4.644s
peak memory: 180mb

That's a 20,000 x 20,000 pyramidal tiff converted to deepzoom in under 10 seconds on a small laptop. It's writing a zip file containing the pyramid, so you can upload to a server immediately. Memory use scales with image width, so it'll do very large images --- I regularly process 250,000 x 250,000 pixel slides.

There's a chapter in the docs introducing dzsave.

For your (my) image, the limiting factor is the size of the pixel cache, which is limited by the setting "MAGICK_AREA_LIMIT". The default of 4GB is not large enough for 72224 x 64080 - that would require a setting of at least 4.4GB - try "MAGICK_AREA_LIMIT=8GB."

If you want to control the impact that ImageMagick has on system RAM and the system page file, then you can limit that using "MAGICK_MEMORY_LIMIT." In truth, there isn't much need to use a large limit there since the fallback location for the pixel cache is mapped memory files, which are on the same order of magnitude of efficiency as the system page file. Try "MAGICK_MEMORY_LIMIT=2GB", to keep the pixel cache out of there (not that it would go there anyways - it is way bigger than 12GB.)

You want the pixel cache to go to mapped memory, so try "MAGICK_MAP_LIMIT=100GB" to take advantage of that space you have. The memory mapped files will end up, not in the system temp directory, but in the directory specified by "MAGICK_TMPDIR".

For extra credit, you also might experiment with the Q8 version, since you don't need 16 bit color channels. You can expect roughly half the disk io with that version.

Good luck! David

The Q8 version uses half the disk space and time to complete a conversion compared to the Q16 version! Also, if you are going to end up breaking up the image into tiles, you can do that in a single step with a command like:

convert.exe" "WRL_15_1A.tif[2]" -crop 14409x15396 +repage -scene 0 "temp\WRL_15_1A%d.tif"

• The "[2]" calls out the third image (the one with the highest resolution.)
• The -crop parameters are 1/4 of the width and height respectively, giving us 16 tiles.
• The +repage sets all of the tiles at origin (0,0)
• The "%d" numbers the files, starting at the # set by "-scene".

Imagemagick has a format for handling large files (mpc). Basically trading disk space for ram. Two files are generated on convert, .mpc and .cache, and you can run imagemagick commands on the smaller .mpc file. These files may only work on your current build of imagemagick, so they aren't suitable for archive.