ActiveMQ: outOfMemoryError: unable to create new native thread
I'm currently load testing the default REST API of activemq. All my 100 users write the same message text to the same destination as fast as they can. After a few seconds i get:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:92)
at org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:85)
at org.apache.activemq.broker.TransportConnector$1.onAccept(TransportConnector.java:211)
at org.apache.activemq.transport.vm.VMTransportServer.connect(VMTransportServer.java:91)
at org.apache.activemq.transport.vm.VMTransportFactory.doCompositeConnect(VMTransportFactory.java:145)
at org.apache.activemq.transport.vm.VMTransportFactory.doConnect(VMTransportFactory.java:53)
at org.apache.activemq.transport.TransportFactory.doConnect(TransportFactory.java:51)
at org.apache.activemq.transport.TransportFactory.connect(TransportFactory.java:80)
at org.apache.activemq.ActiveMQConnectionFactory.createTransport(ActiveMQConnectionFactory.java:243)
at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:258)
at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:230)
at org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:178)
at org.apache.activemq.web.WebClient.getConnection(WebClient.java:246)
at org.apache.activemq.web.WebClient.createSession(WebClient.java:348)
at org.apache.activemq.web.WebClient.getSession(WebClient.java:239)
at org.apache.activemq.web.MessageServletSupport.getDestination(MessageServletSupport.java:319)
at org.apache.activemq.web.MessageServletSupport.getDestination(MessageServletSupport.java:267)
at org.apache.activemq.web.MessageServlet.doPost(MessageServlet.java:110)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:713)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:527)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1216)
at org.apache.activemq.web.SessionFilter.doFilter(SessionFilter.java:45)
at开发者_如何学运维 org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1187)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:421)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:493)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:930)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:358)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:866)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:456)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
at org.eclipse.jetty.server.Server.handle(Server.java:351)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:594)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1059)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:764)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:217)
at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:424)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:506)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
at java.lang.Thread.run(Thread.java:662)
Since I'm already running with 10G heapspace and only simulating 10 REST users I don't understand why activemq is generating 8000 threads. I hope someone can help me fix this.
The reason for this memory leak lies within activemq's connection handling. The default servlet creates one connection for each httpsession (which is a really bad idea). Me loadtesting tool doesn't care about sessions and doesn't send a session-id. So every message that was send is a new connection, is a new thread and after 17.000 threads the jvm is done.
Basicly I'm going to write a new servlet that uses one connection per worker thread. That should limmit the threadnumber to a bare minimum.
精彩评论