SSRS 2005 API Causing Application Pool Crash

My site has started having issues with the application pool crashing while using Reporting Services manager. This is happening at more than one installation. I have seperated both the Reports and ReportServer web sites into their own application pool to single out the issue and have localized the issue to ReportServer as it is the only app pool that is crashing.

This started at the time we implemented the ReportingServices2005 API. Attached below is a sample call we use to retrieve the list of reports in an SSRS folder.

Currently our application uses javascript (Jquery) to make ajax calls to a C# .NET function which makes the SSRS API call.

It seems to happen randomly as the app pool will not crash for days or will crash within hours of restarting. Once this error happens, the app pool rejects all future connections, however the IIS UI does not show that the app pool is stopped. The httperr log shows that the app pool was disabled due to rapid-fail protection. When rapid-fail protection is turned off the crash still occurs and the app pool drops all connections.

We have looked at all logs from app pool as well as reporting services. We have also added the Debug Diagnostics Tool to IIS to retrieve the dump file from the server. Nothing obvious has jumped out at us and we’re now at a loss.

Example Reporting Services API Call:

    [WebMethod(EnableSession = false)]
    [ScriptMethod]
    public List<CatalogItem> GetReportList()
    {
        CatalogItem[] items = null;
        List<CatalogItem> ret = new List<CatalogItem>();
        try
        {
            ReportingService2005 rs = new ReportingService2005();
            rs.Url = ConfigurationManager.AppSettings["Reporting_Services_Url"];
            rs.Credentials = System.Net.CredentialCache.DefaultCredentials;

            // Use the reports path in the app config 
            string path = ConfigurationManager.AppSettings["ReportServer_ReportList_Path"];

            if (path == null || path == "") path = "/"; // send in at least a base root "/"
            items = rs.ListChildren(path, true);

            foreach (CatalogItem item in items)
            {
                if (item.Type == ItemTypeEnum.Folder || item.Type == ItemTypeEnum.Report)
                {
                    if (item.Name != "Search Reports" && item.Name != "My Reports" && item.Name != "Users Folders" && item.Name != "Data Sources")
                        ret.Add(item);
                }
            }
        }
        catch (System.Exception ex)
        {
            Guid? guid = ExceptionPolicy.HandledException(ex);
            // json object to include the guid as well as the exception info, for UI retrieval
            string jsonErrData = AjaxErrorHandling.CreateJsonFormat(guid, ex.Message, ex.GetType().ToString());
            throw new Exception(jsonErrData);
        }
        return ret;
    }

Dump Log Warning:

The following threads in Crash_Rule_Targets_Date_11_03_2010__Time_09_16_14AM_w3wp.exe_ReportServer_PID_5256_371_Manual Dump.dmp are waiting on data to be returned from another server via WinSock The call to WinSock originated from 0x02a6a1c3 and is destined for port 80 at IP address 127.0.0.1

( 62 )1.12% of threads blocked

Ensure that any remote server this application may be calling is functioning properly and there are no network issues between the two servers. If the problem continues, please contact the application vendor for further assistance

Dump Thread in question:

Thread 62 - System ID 3024 Entry point mscorwks!Thread::intermediateThreadProc Create time 11/2/2010 7:21:00 PM Time spent in user mode 0 Days 00:00:00.734 Time spent in kernel mode 0 Days 00:00:00.156

This thread is waiting on data to be returned from another server via WinSock.

The call to WinSock originated from 0x02a6a1c3 and is destined for port 80 at IP address 127.0.0.1

WinDbg Dump access on Thread 62 OS Thread Id: 0xbd0 (62) ESP EIP 2b13edb0 7c82860c [NDirectMethodFrameStandalone: 2b13edb0] System.Net.UnsafeNclNativeMethods+OSSOCK.recv(IntPtr, Byte*, Int32, System.Net.Sockets.SocketFlags) 2b13edc8 7a581b04 System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef) 2b13ee00 7a581a00 System.Net.Sockets.Socket.Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags) 2b13ee20 7a58b854 System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) 2b13ee50 7a58b7bb System.Net.PooledStream.Read(Byte[], Int32, Int32) 2b13ee6c 7a58b4aa System.Net.Connection.SyncRead(System.Net.HttpWebRequest, Boolean, Boolean) 2b13eec0 7a58b30a System.Net.Connection.PollAndRead(System.Net.HttpWebRequest, Boolean) 2b13eed8 7a58b29b System.Net.ConnectStream.PollAndRead(Boolean) 2b13eee4 7a58b132 System.Net.HttpWebRequest.EndWriteHeaders(Boolean) 2b13ef10 7a58b056 System.Net.HttpWebRequest.WriteHeadersCallback(System.Net.WebExceptionStatus, System.Net.ConnectStream, Boolean) 2b13ef24 7a58ac11 System.Net.ConnectStream.WriteHeaders(Boolean) 2b13ef78 7a588f82 System.Net.HttpWebRequest.EndSubmitRequest() 2b13efa4 7a588e9f System.Net.HttpWebRequest.SetRequestSubmitDone(System.Net.ConnectStream) 2b13efb8 7a5888c2 System.Net.Connection.CompleteStartRequest(Boolean, System.Net.HttpWebRequest, System.Net.TriState) 2b13efe8 7a588233 System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest) 2b13f02c 7a587bac System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String) 2b13f064 7a587479 System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint) 2b13f090 7aaaa4d4 System.Net.HttpWebRequest.GetResponse() 2b13f0d0 65dda7e5 System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(System.Net.WebRequest) 2b13f100 65dde895 System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(System.Net.WebRequest) 2b13f104 2f51e76e Microsoft.ReportingServices.UI.Global+RSWebServiceWrapper.GetWebResponse(System.Net.WebRequest) 2b13f114 65e09905 System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(System.String, System.Object[]) 2b13f158 2be6bd5c Microsoft.SqlServer.ReportingServices2005.ReportingService2005.ListSecureMethods() 2b13f164 2f51df12 Microsoft.SqlServer.ReportingServices2005.RSConnection.GetSecureMethods() 2b13f19c 2f51de3a Microsoft.ReportingServices.UI.Global+RSWebServiceWrapper.GetSecureMethods() 2b13f1c4 2f51dd9b Microsoft.SqlServer.ReportingServices2005.RSConnection.IsSecureMethod(System.String) 2b13f1dc 2f51dd13 Microsoft.SqlServer.ReportingServices2005.RSConnection.ValidateConnection() 2b13f204 2f5197d2 Microsoft.ReportingServices.UI.Global.SecureAllAPI() 2b13f20c 2f519720 Microsoft.ReportingServices.UI.ReportingPage.EnsureHttpsLevel(Microsoft.ReportingServices.Diagnostics.HttpsLevel) 2b13f218 2f5191e4 Microsoft.ReportingServices.UI.ReportingPage.ReportingPage_Init(System.Object, System.EventArgs) 2b13f260 02a6b30d [MulticastFrame: 2b13f260] System.EventHandler.Invoke(System.Object, System.EventArgs) 2b13f274 660ab484 System.Web.UI.Control.OnInit(System.EventArgs) 2b13f288 660a72ed System.Web.UI.Page.OnInit(System.EventArgs) 2b13f294 660ab5de System.Web.UI.Control.InitRecursive(System.Web.UI.Control) 2b13f2b4 660a7a3b System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean) 2b13f40c 660a7764 System.Web.UI.Page.ProcessRequest(Boolean, Boolean) 2b13f444 660a7691 System.Web.UI.Page.ProcessRequest() 2b13f47c 660a7626 System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext) 2b13f488 660a7602 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext) 2b13f49c 2f519105 ASP.pages_report_aspx.ProcessRequest(System.Web.HttpContext) 2b13f4a0 660adad6 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 2b13f4d4 6608132c System.Web.HttpApplicat开发者_JS百科ion.ExecuteStep(IExecutionStep, Boolean ByRef) 2b13f514 6608c5c3 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception) 2b13f564 660808ac System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object) 2b13f580 66083e1c System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest) 2b13f5b4 66083ac3 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest) 2b13f5c4 66082c5c System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32) 2b13f7d8 79f9811e [ContextTransitionFrame: 2b13f7d8] 2b13f80c 79f9811e [GCFrame: 2b13f80c] 2b13f968 79f9811e [ComMethodFrame: 2b13f968]

check your db server health, monitor ReportServer db, and your application database. Run the profiler to catch deadlock, locks, transaction time outs etc.

Disable SSRS cache for time, and see if this error happens again.

Anything in the eventlog pertaining to WAS?