c# Performance slowdown while looping/iterating through a long process

OVERVIEW

I am facing performance slowdown while iterating MANY times through a calculator class. Iterations take about 3mn each at the beginning and take longer and longer as the iteration count grows (30mn+/per process). I have to Stop the program/Restart the execution where I left it to come back to normal conditions (3mn/per process).

WHAT I DO

I have a scientific application that tests a set of parameters over a process. For example I have N scenarios (i.e a parameter combination), tested over an experimentation set, that consists in a calculator class that takes the parameters in input, processes them against T possible XP conditions, and stores the output in ORM objects, that are fired to DB after each iteration.

In other words, Each of the N Parameters combination is passed T times trough the calculator.

Parameter combination : Params Set 1, Params Set 2, ...., Params Set  N
Experimental Set      : XP Set 1    , XP Set 2    , ...., XP Set T

So I have NxT combinations, N and T being around 256 each, which give 65000+ iterations.

HOW I DO IT

I have a GUI to fix the parameter sets, and launch Background Workers (one per Parameter combination). Each Backrgound worker loads the first of the T XP sets, executes the current Parameter Set, move to next XP Set, and so on . A report is generated after each single iteration by the calculator (i.e after each Nx/Tx) and an event is fired to populate .NET Linq/SQL ORM objects (AgileFX) and store them into an SQL Server Database.

THE PROBLEM

The process runs fine the first 30mn and then slowly begins to drift, each iteration 开发者_JAVA百科taking longer and longer (Sound like a memory overflow or so...)

HINT

Oddly enough, an experimenter noticed very pertinently that the processing time grows in a linear fashion : +3mn more of the precedent processing time. Which comes down to an arithmetic progression (Tn+1 = Tn + 3mn) I have a 12-Core INTEL and 24GB RAM

A quick suggestion, could you solve your problem through Memoization, avoiding re-calculating what should have been known results?

Also, remember that your garbage collector will not be able to do a garbage collection if you have it will find a reference to the object in some way!

I think I have found one part of the problem, but it did not fix the issue completely :

Objects where sent to the ORM via Delegates registered by a Listener, so each Calculation Thread was still "existing" in the memory even after it has ended. As a colleague stated it : "Even if you move off, If I still have your address in my registers, ror me you still live in the neighborhood."

BTW, performance wizard in VS2010 works a treat. Extremely insightful and useful for monitoring overall memory performance with precision and accuracy.

EDIT : PROBLEM SOLVED
The class responsible for firing background workers was keeping track of some data in a tracker object that kept growing on and on and never flushed, getting bigger and bigger. I've noticed this by closely tracking memory usage per object in VS 2010 Performance Wizard.
I advice having a clear view of objects lifecycle, and memory usage, although it can get tough when the application is big and complex.