IIS6 Application Pool Crash

During recent stress and volume testing, we realized that after 30minutes, all uses gets disconnected from the website. After event logging, it came to our attention that the application pool crashes. Doing some google investigation, apparently in most causes this is due to unhandled exceptions.

SO when the application crashes, the following exception details are displayed:

An unhandled exception occurred and the process was terminated.

Application ID: DefaultDomain

Process ID: 7852

Exception: System.Runtime.Serialization.SerializationException

Message: Type 'FuseFarm.FrameworkException' in Assembly 'FuseFarm, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' is not marked as serializable.

StackTrace:    at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjec开发者_如何转开发tInfoInit, IFormatterConverter converter, ObjectWriter objectWriter)
   at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.Serialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter)
   at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header[] inHeaders, __BinaryWriter serWriter, Boolean fCheck)
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header[] headers, Boolean fCheck)
   at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.SerializeObject(Object obj, MemoryStream stm)
   at System.AppDomain.Serialize(Object o)
   at System.AppDomain.MarshalObject(Object o)

I have no idea why it's trying to Serialize FrameworkException, and I can't see in the code where this is being done either. But I do see several parts of code where

new FrameworkException(exData, "ContractComposition.SubmitContract");

is being called, but not being handled. After checking the global.asax.cs, the following is happening:

protected void Application_Error(object sender, EventArgs e)
{
    ILog log = LogManager.GetLogger(typeof(Global));
    string environmentName = WebConfigurationManager.AppSettings["EnvironmentName"];
    if (!String.IsNullOrEmpty(environmentName) && (environmentName == "DEMO" || environmentName == "LIVE"))
    {
        Exception currentException = Server.GetLastError().GetBaseException();
        Session["errorMessage"] = currentException.Message;
        Session["errorSource"] = currentException.Source;
        Session["errorTrace"] = currentException.StackTrace;
        log.Error(currentException.Message, currentException);
        if (currentException != null)
        {
            Session["error"] = currentException.GetType().Name;
            switch (currentException.GetType().ToString())
            {
                case "System.ApplicationException":
                case "FuseFarm.FrameworkException":
                    break;
                default:
                    new FrameworkException(currentException.Message + "\n" + currentException.StackTrace, currentException.Source, currentException);
                    break;
            }

        }
        Server.Transfer("~/error.aspx");
    }
}

Throwing a new exception in Application_Error... This doesn't seem right? Who and what will handle this error if it's thrown at this point?

It's Serializing FrameworkException because it is trying to go across an AppDomain boundary.

All objects that go across an AppDomain must be serialized, and an exception is no different.

You can consider it a bug when an Exception does not properly implement serialization.

I don't believe that your error handler is what is the source of the problem. It's hard to tell given that stack trace - a full memory dump would yield better information.

Your best bet is to just properly make the exception serializable.

This may not completely resolve your issue - boiled down, you will still be throwing exceptions. Hopefully once that is corrected you will see the real cause of the problem.

If an exception is thrown in Application_Error, your app pool will crash, just like you're seeing. Based on the code you've shown, I would conclude that the error logger referred to in the line:

log.Error(currentException.Message, currentException);

is attempting to serialize exceptions that are passed into it. When currentException ends up being of type FuseFarm.FrameworkException, the error handler crashes on that line and your app pool shuts down.

If this is the case, you'll need to either mark FuseFarm.FrameworkException as Serializable, change the logger to not attempt to serialize objects, or put a try/catch block in the Application_error handler and do something else with those exceptions if you wish the app pool to keep going.