Is it possible to create smaller memory-mapped views on a 2GB+ file?

I have written a C++ class for Windows and Linux that creates a memory-mapped view for an file of arbitrary size n. The code for the class constructor can be seen here. I am currently testing the code on Windows 32 bit XP.

I have found for file sizes 0 < n <= 1.7GB , the constructor returns a valid pointer to a memory-mapped view. However, for a file size >= 2 GB, MapViewOfFile returns a NULL value and an error code of 8, "Not enough storage is available to process this command". Evidently, Windows cannot find an available address space of size 2 GB in the process.

Therefore, I may need to modify the class constructor to create a set of smaller memory-mapped views totaling >= 2GB bytes && < 2 ^ 32 - 1 bytes. The other requirement is to create a mapping between each of the smaller memory-mapped views and a randomly accessed address in the process' address space.

Previously, I used the following code for random access:

char* KeyArray;

try {
    mmapFile = new cMemoryMappedFile(n);
}
catch (cException e)
{
    throw;
}

KeyArray = (char *)(mmapFile->GetPointer());
KeyArray[i开发者_JAVA百科] = ...

How should I modify the class to handle these requirements?

What you want can be achieved using repaging, see how it is done in boost.iostreams here.

You only have 2GB (or 3GB, with some tweaks) of user process space on a 32-bit OS. Period. That is a hard limitation, and no amount of creating many smaller mappings can get around that. You will need to shift your mapping around in order to access the different parts of the file. But it will still be faster than seeking, reading, and writing.

I can't see your pastebin link, but I can suggest a simple solution with a c++ class declaration. I think the implementation should be obvious from the comments:

class ShiftingMemMap
{
public:
    // constructs a dynamically shifting memory map of a file...
    ShiftingMemMap ( const char* fileName, size_t view_size = 4096 );

    // retrieve/set a byte at the given file offset. If the offset is not currently in-view,
    // shift the view to encompass the offset. The reference should not be stored for later
    // access because the view may need to shift again...
    byte& operator [] ( unsigned int_64_t offset );

private:
    int_64_t current_offset;
    size_t current_size;
};

All that being said, you could write a class that returns multiple views of a file to allow saving a reference for later and also editing different parts of the file simultaneously without having to shift the view back and forth repeatedly.

class MemMap
{
public:
    MemMap ( const char* filename );

    shared_ptr<MemMapView> View ( unsigned int_64_t offset, size_t size = 4096 );
};

class MemMapView
{
public:
    char& operator[] ( size_t offset );
};

This is not going to work. You simply cannot use all of the 4 GiB of address space on 32 bit Windows. Redesign your access to the array to map just small views of the large file.

Yes, it is possible to write a C++ class for Windows,Solaris Unix, Linux that creates a memory-mapped view for an file of arbitrary size n. We just finished building two version of class, one using STL and the other one which my boss wrote which does not use STL. Both versions are better than using the heap when the size of the memory allocation is 1 gigabyte or more on 32 bit Windows, Linux, Solaris Unix. Both of these version are also compatible with 64 bit Windows, Linux, Solaris Unix. If you want to find out more details about this, please email me at frankchang91@gmail.com. Massachusetts, USA. these 2 versions were independently designed and may become part of a data fusion US patent filing.