How to keep windows from paging block of memory

We are working on a Vista/Windows 7 application that will be running in 64开发者_C百科 bit mode using VS2008/C++. We will be needing to cache hundreds of 2-3 mb blobs of data in RAM for performance reasons up to some memory limit. Our usage profile is such that we cannot read the data in fast enough if it is all on the the disk. Cached Memory usage will be larger than 1gb memory used. For this to work well, we need to ensure that Windows does not page this memory out as it will defeat the purpose of why we are doing this.

I've done a fair amount of research and cannot find documentation that states exactly how to do this. I've seen several references that infer memory mapped files work this way. Is there an expert who can clarify this for me?

I'm aware there are other programs that we could adapt to do this, for example, splitting the blobs and loading into memcache or inmemory databases, but they all have too many problems with performance or code complexity.

Suggestions?

You can use VirtualLock. However, you'll surely hit the quota with the amount you're talking about. Given that you should never run any other code on this machine, you'll be better off just disabling the paging file. Control Panel + System + Advanced.

From user mode, you can't (EDIT: At least for the sizes you're talking about). User mode allocations all come down to either the VirtualAlloc API (On top of which the GlobalAlloc/LocalAlloc/C Runtime's functions are written) or the Memory Mapped File API. Neither API supports this, and therefore it's impossible to obtain on Win32. It is possible from whithin Kernel Mode, but somehow I suspect this is a user-mode application :)

Note that the memory manager is not going to decide to page your RAM without good reason to do so.

Now, you could of course, if you control the machine completely (this is for internal use or something) disable the pagefile on the machine in question, but that does not seem to solve your problem.

perhaps the answer? (from a VMWARE tutorial)

To edit the Registry and disable paging kernel-mode stacks

1. Click Start > Run and type regedit.
2. In the left pane of the Registry Editor, navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager.
3. In the right pane, right‐click GlobalFlag and select Modify.
4. With Base Hexadecimal, type value 80000, which corresponds to FLG_DISABLE_PAGE_KERNEL_STACKS.
5. Click OK and exit the Registry Editor.
6. Reboot the guest system for this change to take effect.

hope it helps

It's possible! You can force pages to be locked in memory from a user mode app by allocating them using AWE (Address Windowing Extensions) VirtualAlloc + AllocatePhysicalPages + MapPhysicalPages.

Note: I have read that you can use the AWE APIs from either a 32-bit or 64-bit app also, but I've only tried with 32-bit app. (Of course since it's AWE you can manually remap memory to access > 2GB RAM.)

Note: You have to first have seLockMemoryPrivilege. (Which seems to require the app to run as Administrator in my testing so far.)

Note: Using AWE implies some limitations on what you can do with those particular pages of memory, e.g. no VirtualProtect().