Matlab runtime error for bigger data-arrays passing to mex function

I wrote a mex function and when I call it for some smaller arrays, everything works fine. When passing a bigger array, I get the following runtime error.

http://img192.imageshack.us/img192/6262/zwischenablage02xt.jpg

I use Matlab 2010b, 32bit on Win7 Pro x64. I have the classic windows theme (XP style) activated since I read about this as a problem here: http://www.edaboard.com/thread20634.html

I also installed the latest Java runtime version. I use CUDA in my mexfile and have the CUDA toolkit and SDK 3.2 32bit installed and a device driver (forceware)... I just don't know what could be the problem... Memory is enhough there:

>> memory
Maxim开发者_JAVA百科um possible array:            2046 MB (2.146e+009 bytes) *
Memory available for all arrays:   3485 MB (3.655e+009 bytes) **
Memory used by MATLAB:              308 MB (3.225e+008 bytes)
Physical Memory (RAM):             8190 MB (8.588e+009 bytes)

*  Limited by contiguous virtual address space available.
** Limited by virtual address space available.

I really investigated it hard but can't come up with another idea :( Any clue about it? Thanks!