Recommendations for IPC between parent and child processes in .NET?

My .NET program needs to run an algorithm that makes heavy use of 3rd party libraries (32-bit), most of which are unmanaged code. I want to drive the CPU as hard as I can, so the code runs several threads in parallel to divide up the work.

I find that running all these threads simultaneously results in temporary memory spikes, causing the process' virtual memory size to approach the 2 GB limit. This memory is released back pretty quickly, but occasionally if enough threads enter the wrong sections of code at once, the process crosses the "red line" and either the unmanaged code or the .NE开发者_如何学PythonT code encounters an out of memory error. I can throttle back the number of threads but then my CPU usage is not as high as I would like.

I am thinking of creating worker processes rather than worker threads to help avoid the out of memory errors, since doing so would give each thread of execution its own 2 GB of virtual address space (my box has lots of RAM). I am wondering what are the best/easiest methods to communicate the input and output between the processes in .NET?

The file system is an obvious choice. I am used to shared memory, named pipes, and such from my UNIX background. Is there a Windows or .NET specific mechanism I should use?

IPC in .NET 3.0+ can be done with WCF and named pipes binding. The most obvious benefit is that you can split your heavy tasks across machines by simply switching the binding. More on this here.

Regarding memory footprint you may try to get your 3rd party libraries in 64-bit and run your stuff on 64-bit OS to get more memory available to your process.

If you use WPF, many common forms of IPC are available. This can be done in such a way that config file changes are all that is required, and leaves room for communicating across machine boundaries as well.

Yes you are right. Breaking into several processes as opposed to threads will help, since every process will have its own 2GB virtual space (on 32-bit system).

Since you have lots of RAM, One other thing you can do (without need to have multiple processes) is use PAE (physical address extension) and then your process can address upwards of 8 GB . See here

As others have suggested you can use .NET remoting (if both client and server are .NET, can be local in same machine or remote).

PS: this answer is late, may be your requirement is done with, but i post it here for later use of this thread by other people.