How can I create a numpy .npy file in place on disk?

Is it possible to create an .npy file without allocating the corresponding array in memory first?

I need to create and work with a large numpy array, too big to create in memory. Numpy supports memory mapping, but as far as I can see my options are either:

1. Create a memmapped file using numpy.memmap. This creates the file directly on disk without allocating memory, but doesn't store the metadata, so when I re-map the file later I need to know its dtype, shape, etc. In the following, notice that not specifying the shape results in the memmap being interpreted as flat array:

   In [7开发者_JS百科7]: x=memmap('/tmp/x', int, 'w+', shape=(3,3))
   
   
   In [78]: x
   Out[78]: 
   memmap([[0, 0, 0],
          [0, 0, 0],
          [0, 0, 0]])
   
   
   In [79]: y=memmap('/tmp/x', int, 'r')
   
   
   In [80]: y
   Out[80]: memmap([0, 0, 0, 0, 0, 0, 0, 0, 0])
   

2. Create an array in memory, save it using numpy.save, after which it can be loaded in memmapped mode. This records metadata with the array data on disk, but requires that memory be allocated for the entire array at least once.

I had the same question and was disappointed when I read Sven's reply. Seems as though numpy would be missing out on some key functionality if you couldn't have a huge array on file and work on little pieces of it at a time. Your case seems to be close to one of the use cases in the origional rational for making the .npy format (see: http://svn.scipy.org/svn/numpy/trunk/doc/neps/npy-format.txt).

I then ran into numpy.lib.format, which seems to be full useful goodies. I have no idea why this functionality is not available from the numpy root package. The key advantage over HDF5 is that this ships with numpy.

>>> print numpy.lib.format.open_memmap.__doc__

"""
Open a .npy file as a memory-mapped array.

This may be used to read an existing file or create a new one.

Parameters
----------
filename : str
    The name of the file on disk. This may not be a filelike object.
mode : str, optional
    The mode to open the file with. In addition to the standard file modes,
    'c' is also accepted to mean "copy on write". See `numpy.memmap` for
    the available mode strings.
dtype : dtype, optional
    The data type of the array if we are creating a new file in "write"
    mode.
shape : tuple of int, optional
    The shape of the array if we are creating a new file in "write"
    mode.
fortran_order : bool, optional
    Whether the array should be Fortran-contiguous (True) or
    C-contiguous (False) if we are creating a new file in "write" mode.
version : tuple of int (major, minor)
    If the mode is a "write" mode, then this is the version of the file
    format used to create the file.

Returns
-------
marray : numpy.memmap
    The memory-mapped array.

Raises
------
ValueError
    If the data or the mode is invalid.
IOError
    If the file is not found or cannot be opened correctly.

See Also
--------
numpy.memmap
"""

As you have found out yourself, NumPy is mainly targetted at handling data in memory. There are different libraries for handling data on disk, the one most commonly used today probably being HDF5. I suggest having a look at h5py, an excellent Python wrapper for the HDF5 libraries. It is designed to be used together with NumPy, and its interface is easy to learn if you already know NumPy. To get an impression how it tackles your problem, read the documentation of Datasets.

For the sake of completeness I should mention PyTables, which seems to be the "standard" way of handling large datasets in Python. I did not use it because h5py appealed more to me. Both libraries have FAQ entries defining their scope against the other one.