Hashtable - out of memory

I am using Hashtable in my c# application. I am loading millions of key into, but after the application exceed the 3,7GB of RAM it gives me an "out of memor开发者_运维百科y" exception.

I use x64 operation system, and the computer has 16GB of ram. I was thinking about maybe this could be an x86 limitation. I changed the build type to x64 but I still get the error.

Is there a maximum memory size for an object in .net? Can I do something to use all the memory?

Thanks, Andrew

Take a look on this previous answer: .NET Max Memory Use 2GB even for x64 Assemblies

The 2GB limit applies to each object individually. The total memory used for all objects can exceed 2GB.

Use a Dictionary<,> instead of HashTable. In a HashTable both the key and value are objects, so if they are value types they will be boxed. A Dictionary can have value types as key and/or value, which uses less memory. If you for example use an int as key, each will use 28 bytes in a HashTable while they only use 4 bytes in a Dictionary.

If both the key and value are value types and use less than 8 bytes the Dictionary will be able to hold more items than the HashTable.

This article states that .NET limits the size of a single object to two gigabyte.

I found this page really useful as I was having exactly the same problem in VB .net. I had created some very large hashtables and my application was throwing an out of memory exception when trying to create them.

I tried "Guffa's" idea of using a Dictionary on a smaller example hashtable and the resulting dictionary actually increased in size!

His suggestion got me thinking about what I was actually writing to the Hashtable. So I looked at this page and decided I did not really need to be storing data of type Long, an Integer was more than sufficient. I know I'm probably telling all you gurus out there what is blindingly obvious but I was able to reduce the resulting serialized hastables (12 of them in total) from 81.9Mb to 69.4Mb simply by declaring the variables as integers instead of Longs.

This did not actually resolve my problem but will certainly improve the time taken to load them back into memory.

The problem has nothing to do with the amount of physical RAM in your system, or whether is x64 compiled or not. Nor is it now true that .NET must limit objects to 2GB. (You may want to investigate .NET 4.5's Large Object Heap for more information on this.) The reason you are getting "Out of memory" is because the process cannot map a contiguous section of memory in the size you requested.

Eric Lippert has an excellent blog article on this very topic, and I notice that he commented on this question, but unfortunately did not answer it.

So here's the answer: accept the fact that the memory size you seek with this implementation is not readily available, and solve your problem by breaking it down into smaller memory chunks.

I had this same problem and had to take the same approach. Working with jagged arrays may help, or breaking your data into multiple logical components and create a series of smaller HashTables.