CLR memory consumption problem
One of issues with CLR is it's extremely bad behavior in the lack of RAM开发者_开发问答 (when some of memory of managed process gets paged out, this leads to the total freeze of the whole system, even Ctrl-Alt-Del screen can't be accessed. I assume the reason is GC which tries to build graph of reachable objects and tries to scan all memory of the process, causing massive page in/page out operations).
This makes a problem for my .NET program, coz it can consume lots of RAM when input data is huge.I'd prefer to show a "not enough memory" message to the user, rather than completely hang his system ^_^
Is there any way to achieve this?With MemoryFailPoint
, you can tell .NET you are going to need a certain amount of memory. The problem though is that even this system includes swap space.
I believe that it is very difficult to achieve what you want to achieve here. Say, you would use some system indicators and performance indicators to find out how much physical memory is available and based on that, perform some tasks. If after you've completed this check a different process comes in a grabs physical memory, your original calculations do not apply anymore and some of your memory is going to be pushed to swap.
I do have a suggestion. You could have a configuration setting with the maximum allowed amount of memory to be used by the application? With this, you can:
Try to figure out how much resources your application consumes based on e.g. a network connection (if your application is a network server) and throttle the number of connections based on the maximum memory consumption, or
You could have a second thread running that checks ever 10 seconds or minute of so the total memory consumption with
GC.GetTotalMemory()
and start rejecting connections (again, if your application is a network server) once you get to that maximum.
This should be a configuration setting instead of e.g. the amount of physical memory available, because you do not know what other applications are running on the machine.
There are system calls to get the size of a process and the account of RAM on a machine that may help.
There used to be a lot of research on writting GCs that could cope with paging, but I expect they will never ship as RAM is getter so big these days. (The basic ideal was not to collect any objects that are paged out, and to try to collect all objects on page just before the OS paged it out. But you need to know all objects that may be pointed to by a paged out object.)
You may be able to use arrays of structs and then pass about the indexes so as to reduce the number of objects you have, and hence the number of pointers the GC have to follow. This is only worthwhile if you have a LOT of data of the same type you need in RAM.
In Windows, you can use a Job Object to put a hard limit on the amount of virtual memory that a process can allocate. The CLR will automatically garbage collect when it hits this limit, and throw an OutOfMemoryException if it cannot free enough space. You can also limit the working set of a process instead of the virtual memory. This allows the process to allocate as much as it likes but it will get swapped out instead of consuming RAM and the system won't hang. I've used Job Objects successfully for exactly this purpose (keeping the machine responsive while running large jobs).
To create job objects from .NET, you have to use PInvoke calls to the Win32 API. This article on CodeProject explains the procedure.
Have IIS host the process and use the memory controls in IIS to limit the amount of memory used. Hopefully this will force GC more often and minimize the need for a long-running GC session. I would imagine that this behavior is due to how your app is using memory from a generational aspect, as in a large number of small objects that are 1) long-lived and 2) getting moved around in memory.
Other links to similiar .NET/CLR memory control issues and solutions
Configure .NET CLR RAM usage
Restricting .Net CLR memory usage
Virtual and Physical Memory / OutOfMemoryException
Preventing OutOfMemoryException with GC.AddMemoryPressure()?
Force garbage collection of arrays, C#
examples of garbage collection bottlenecks
Suppressing C# garbage collection
how to profile .net garbage collector?
In case you are using the Large Object Heap (LOH)
.NET Collections and the Large Object Heap (LOH)
Large Object Heap Fragmentation
精彩评论