Intermittetly NPE during parseParameters in Tomcat's Request

We have intermittet NPE during parseParemeters in org.apache.catalina.connector.Request. The more users are online, the more this NPE happens. After a JBoss restart, the NPEs disappear for a while. Within 24 hours we receive between one and over 400 of those NPE. It does not matter which service is called. Any service request can end in this NPE.

java.lang.NullPointerException
        at org.apache.catalina.connector.Request.parseParameters(Request.java:2517)
        at org.apache.catalina.connector.Request.getParameterNames(Request.java:1102)
        at org.apache.catalina.connector.Request.getParameterMap(Request.java:1082)
        at org.apache.catalina.connector.RequestFacade.getParameterMap(RequestFacade.java:414)
        at javax.servlet.ServletRequestWrapper.getParameterMap(ServletRequestWrapper.java:166)
        at org.jboss.seam.mock.MockExternalContext.getRequestParameterValuesMap(MockExternalContext.java:307)
        at org.jboss.seam.faces.Parameters.getRequestParameters(Parameters.java:61)
        at org.jboss.seam.Component.injectParameters(Component.java:1586)
        at org.jboss.seam.Component.inject(Component.java:1556)
        at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:61)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.transaction.TransactionInterceptor$1.work(TransactionInterceptor.java:97)
        at org.jboss.seam.util.Work.workInTransaction(Work.java:61)
        at org.jboss.seam.transaction.TransactionInterceptor.aroundInvoke(TransactionInterceptor.java:91)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.security.SecurityInterceptor.aroundInvoke(SecurityInterceptor.java:163)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at ExceptionInterceptor.aroundInvoke(ExceptionInterceptor.java:51)
        at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
        at org.jboss.seam.intercept.Interceptor.aroundInvoke(Interceptor.java:187)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:72)
        at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
        at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:185)
        at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:103)
        at TaskService_$$_javassist_seam_7.getNumberOfUpdatedTasks(TaskService_$$_javassist_seam_7.java)
        at sun.reflect.GeneratedMethodAccessor319.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.seam.remoting.gwt.GWTToSeamAdapter.callWebRemoteMethod(GWTToSeamAdapter.java:100)
        at org.jboss.seam.remoting.gwt.GWTService.RPC_invokeAndEncodeResponse(GWTService.java:550)
        at org.jboss.seam.remoting.gwt.GWTService.processCall(GWTService.java:206)
        at org.jboss.seam.remoting.gwt.GWTService$1.process(GWTService.java:120)
        at org.jboss.seam.servlet.ContextualHttpServletRequest.run(ContextualHttpServletRequest.java:53)
        at org.jboss.seam.remoting.gwt.GWTService.getResource(GWTService.java:105)
        at org.jboss.seam.servlet.SeamResourceServlet.service(SeamResourceServlet.java:80)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
        at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:60)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:40)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73)
        at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
        at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
        at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
    开发者_如何学JAVA    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)
        at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:619)

We use JBoss AS 5.1.0.GA, Seam 2.2.0.GA, and GWT 2.0.3. The JBoss receives the request from Apache 2 via mod_jk. The provided line number (Request.java:2517) indicates that the method of the request is null although the logs of, firebug (client), Apache, and mod_jk show that the method is POST.

Currently, we neither have an idea what may be the root-cause of the NPE nor how we could make a workaround. We are speculating whether the problem has something to do with:

• Garbage collection (JBoss is started with -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000)
• Request recycling in Tomcat
• Filter chain recycling in Tomcat
• mod_jk balancing

What can we do to find the cause of this problem? Is there any possible fix to this problem?

Any help or suggestion is highly appreciated.

Thanks!

--

We had luck and were able to debug the stack trace during NPE. We found out, that the request objects in MockExternalContext do not always match the request objects that the SeamResourceServlet receives. Sometimes the request object in MockExternalContext is new and contains a fresh instance of org.apache.coyote.Request with all fields set to null. If the request can be processed, the request objects received by SeamResourceServlet are identical to those in MockExternalContext.

Can any Seam expert please help us and tell when and where in the above stack trace the MockExternalContext that is used by org.jboss.seam.faces.Parameters is created? Under which circumstances does Seam initialize the MockExternalContext with a fresh request object rather than with the one provided by SeamResourceServlet?

I have cross-posted this problem in the Seam forum.

--

Update

In the mean time we found the reason for the NPEs:

Since we use GWT on the client-side, all client-server communication is done via GWT-RPC asynchronously. Very rarely, a logout call outruns another still processing RPC. The logout call invalidates the session, so the other RPC cannot complete normally which leads to an Exception during ServletLifecycle.endRequest(request); inside the ContextualHttpServletRequest. This Exception is handled by Seam's ExceptionFilter. Unfortunately, the ExceptionFilter can also not complete normally because of the invalidated Session leading to the following error:

ERROR    [Seam Resource Servlet].error: Servlet.service() for servlet Seam Resource Servlet threw exception
java.lang.IllegalStateException: Cannot create a session after the response has been committed
        at org.apache.catalina.connector.Request.doGetSession(Request.java:2338)
        at org.apache.catalina.connector.Request.getSession(Request.java:2094)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:833)
        at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216)
        at org.jboss.seam.mock.MockExternalContext.getSessionMap(MockExternalContext.java:357)
        at org.jboss.seam.contexts.FacesLifecycle.beginExceptionRecovery(FacesLifecycle.java:86)
        at org.jboss.seam.web.ExceptionFilter.endWebRequestAfterException(ExceptionFilter.java:96)
        at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:70)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:40)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73)
        at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
        at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
        at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)
        at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:619)

The MockExternalContext that is created in ExceptionFilter, "somehow" stays in the application context and is "sometimes" used to process requests. It even survives redeployments of our applications, so that we have to restart JBoss to get rid of the NPEs.

Thank you VERY VERY MUCH for this post. This bug is our nightmare for a few weeks. Our config is: GWT 2.0.4, Seam 2.2.1 CR2, JBoss AS 5.1.0. We patched logout mechanism in our app, but it still returns (very rarely, though). Now it seems that it occurs when a transaction lasts too long in EJB tier. Now we're ready to get a rid of Seam, it causes more problems than we can handle.

UPDATE: This strange bug disappeared completely when scope of "ServiceImpl" component was changed (just removed it) from "SESSION" to default. The annotation @BypassInterceptors was also added. Meanwhile I prepared replacement for Seam-GWT bridge for our app. It's Guice + gwt-dispatch. Very fast and reliable solution.