Out of Memory Exception while retrieving large data using web services

I am facing Out of Memory Exception while retrieving large data using web services. it works fine f开发者_高级运维or small amount of data. now in production using large amount of data more than 3 to 4 GB, facing Out of Memory Exception. here we use xml serialization and deserialization.

any help would be appreciated.

I have spent a while looking into out of memory issues while processing XML. Here are a few hints and suggestions:

1) Maximum process size on the 32bit machine can be 2GB and if a startup switch specific, 3GB. So it is a must that 64bit kit is used.

2) Run perfmon.exe to collect samples and see how big memory usage gets for your process before crashing. If you run on 64bit, make sure it is compiled for 64bit otherwise it will run it using WOW and in 32 bit mode.

3) In .NET 1.1, loading a 100MB XML would take 600-800MB of space if processed using XmlDocument. In .NET 2.0 and later it is 3-4 times but still considerable. Avoid XmlDocument when you can.

4) This I think can be your problem: avoid recreating your serializers. .NET actually spits out code and compiles them into assemblies which cannot be unloaded as you know and causes memory leak which is known issue and happens with XSLT transform as well. Always cache your serializers.

The problem I kind of alluded to in the comment is the fact that the runtime cannot allocate a big enough contiguous block of memory.

Eric Lippert has a great blog post on this problem: http://blogs.msdn.com/b/ericlippert/archive/2009/06/08/out-of-memory-does-not-refer-to-physical-memory.aspx