After some time of work, IIS/ASP.Net fails to read the settings from the web.config file of WCF web service

We have an ASP.Net web application running on IIS7 communicating with WCF web service which handles the requests arriving from this web application.

After some time of work (it may be a week or just several days), the WCF web service starts to fail reading the settings in its web.config file, having logged the fol开发者_如何转开发lowing exception(s):

System.NullReferenceException

ABC.Communication.DataContract.RefreshRequest System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Xml.XmlNode.RemoveChild(XmlNode oldChild)
   at System.Xml.XmlNode.RemoveAll()
   at System.Xml.XmlElement.set_InnerXml(String value)
   at System.Configuration.LocalFileSettingsProvider.XmlEscaper.Unescape(String escapedString)
   at System.Configuration.LocalFileSettingsProvider.GetPropertyValues(SettingsContext context, SettingsPropertyCollection properties)
   at System.Configuration.SettingsBase.GetPropertiesFromProvider(SettingsProvider provider)
   at System.Configuration.SettingsBase.GetPropertyValueByName(String propertyName)
   at System.Configuration.SettingsBase.get_Item(String propertyName)
   at System.Configuration.ApplicationSettingsBase.GetPropertyValue(String propertyName)
   at System.Configuration.ApplicationSettingsBase.get_Item(String propertyName)
   at ABC.Communication.DataContract.Request.ValidateIpAddress() in C:\Project\ABC.Server\ABC.Communication\DataContract\Request.Validation.cs:line 955
   at ABC.Communication.DataContract.RefreshRequestBase.Validate() in C:\Project\ABC.Server\ABC.Communication\DataContract\RefreshRequestBase.cs:line 46
   at ABC.Communication.DataContract.RefreshRequest.Validate() in C:\Project\ABC.Server\ABC.Communication\DataContract\RefreshRequest.cs:line 30
   at ABC.Communication.ClientCommunicationService.HandleRequest(Request request) in C:\Project\ABC.Server\ABC.Communication\ClientCommunicationService.cs:line 116

or System.ArgumentException

ABC.Communication.DataContract.RefreshRequest System.ArgumentException: The node to be removed is not a child of this node.
   at System.Xml.XmlNode.RemoveChild(XmlNode oldChild)
   at System.Xml.XmlNode.RemoveAll()
   at System.Xml.XmlElement.set_InnerXml(String value)
   at System.Configuration.LocalFileSettingsProvider.XmlEscaper.Unescape(String escapedString)
   at System.Configuration.LocalFileSettingsProvider.GetPropertyValues(SettingsContext context, SettingsPropertyCollection properties)
   at System.Configuration.SettingsBase.GetPropertiesFromProvider(SettingsProvider provider)
   at System.Configuration.SettingsBase.GetPropertyValueByName(String propertyName)
   at System.Configuration.SettingsBase.get_Item(String propertyName)
   at System.Configuration.ApplicationSettingsBase.GetPropertyValue(String propertyName)
   at System.Configuration.ApplicationSettingsBase.get_Item(String propertyName)
   at ABC.Communication.DataContract.Request.ValidateIpAddress() in C:\Project\ABC.Server\ABC.Communication\DataContract\Request.Validation.cs:line 955
   at ABC.Communication.DataContract.RefreshRequestBase.Validate() in C:\Project\ABC.Server\ABC.Communication\DataContract\RefreshRequestBase.cs:line 46
   at ABC.Communication.DataContract.DataContracts.RefreshRequest.Validate() in C:\Project\ABC.Server\ABC.Communication\DataContract\RefreshRequest.cs:line 30
   at ABC.Communication.ClientCommunicationService.HandleRequest(Request request) in C:\Project\ABC.Server\ABC.Communication\ClientCommunicationService.cs:line 116

or System.Configuration.SettingsPropertyNotFoundException:

ABC.Communication.DataContract.RefreshRequest System.Configuration.SettingsPropertyNotFoundException: The settings property 'ValidateIpConnections' was not found.
   at System.Configuration.SettingsBase.GetPropertyValueByName(String propertyName)
   at System.Configuration.SettingsBase.get_Item(String propertyName)
   at System.Configuration.ApplicationSettingsBase.GetPropertyValue(String propertyName)
   at System.Configuration.ApplicationSettingsBase.get_Item(String propertyName)
   at ABC.Communication.DataContract.Request.ValidateIpAddress() in C:\Project\ABC.Server\ABC.Communication\DataContract\Request.Validation.cs:line 955
   at ABC.Communication.DataContract.RefreshRequestBase.Validate() in C:\Project\ABC.Server\ABC.Communication\DataContract\RefreshRequestBase.cs:line 46
   at ABC.Communication.DataContract.RefreshRequest.Validate() in C:\Project\ABC.Server\ABC.Communication\DataContract\RefreshRequest.cs:line 30
   at ABC.Communication.ClientCommunicationService.HandleRequest(Request request) in C:\Project\ABC.Server\ABC.Communication\ClientCommunicationService.cs:line 116

The code which throws this exception is quite trivial:

private void ValidateIpAddress() {
    // Load custom settings
    CustomSettings settings = GetCustomSettings();

    if (!settings.ValidateIpConnections)   // here the exception is thrown!
        return;

    // the rest of code never reached...
}

Access to the instance of custom settings is implemented now in the top parent class Request as shown below:

public abstract partial class Request
{
    /// <summary>
    /// Custom settings.
    /// </summary>
    private static readonly CustomSettings customSettings;

    /// <summary>
    /// .cctor().
    /// </summary>
    static Request()
    {
        customSettings = new CustomSettings();
    }

    /// <summary>
    /// Get custom settings.
    /// </summary>
    /// <returns>Custom settings.</returns>
    internal static CustomSettings GetCustomSettings()
    {
        return customSettings;
    }

    // ....
}

The issue occurs not only for this specific setting ValidateIpConnections but also for other settings in the web.config file of the WCF service.

It is not clear at all what could be the reason of such issue, it is not reproducible constantly and stably. It does not seem to depend directly on the load/the number of users/open sessions. Sometimes the site is working smoothly for several weeks, then the issue occurs one time a week and then it occurs several days in a row. Possibly, it is related to recycling of some worker process in the application pool.

We used to encounter it on Windows Server 2003 (IIS 6.0) and now get it on Windows Server 2008 with IIS 7.0 so it does not seem to be directly related to the version of IIS.

I suspect there could be an issue with sharing access to the instance of CustomSettings class via readonly static member of Request for all the child classes which is initialized only once.

Any help/suggestion would be greatly appreciated.

I think you can avoid using the custom settings class because the web.config is cached the first time it is accessed. See

ASP.NET: Where/how is web.config cached?

So there's no need to do your own caching and then you can rule that one out.

We had a similar issue on one of our production servers and it turned out that the virus scanner was scanning the web.config file, which led to an application pool recycle.

I would recommend

• Removing the static class and directly use the .net configuration class
• Adjusting the virus scanner that it does not scan the folder where the application is in
• Looking at the event log at the times when the exceptions happen to see if there are other strange things happening
• Setting up asp.net monitoring (ASP.NET Counters > Application restarts): http://msdn.microsoft.com/en-us/library/ms972959.aspx and http://blogs.msdn.com/b/johan/archive/2008/02/18/monitoring-application-pool-and-application-restarts.aspx