How can I monitor/log Tomcat's thread pool?

I have a Tomcat installation where I suspect the thread pool may be decreasing over time due to threads not being properly released. I get an error in catalina.out when maxthreads is reached, but I would like to log the number of threads in use to 开发者_如何学运维a file every five minutes so I can verify this hypothesis. Would anyone please be able to advise how this can be be done?

Also in this installation there is no Tomcat manager, it appears whoever did the original installation deleted the manager webapp for some reason. I'm not sure if manager would be able to do the above or if I can reinstall it without damaging the existing installation? All I really want to do is keep track of the thread pool.

Also, I noticed that maxthreads for Tomcat is 200, but the max number of concurrent connections for Apache is lower (Apache is using mod_proxy and mod_proxy_ajp (AJP 1.3) to feed Tomcat). That seems wrong too, what is the correct relationship between these numbers?

Any help much appreciated :D

Update: Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.

Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.

Direct JMX access

Try adding this to catalina.sh/bat:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=5005
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

UPDATE: Alex P suggest that the following settings might also be required in some situations:

-Dcom.sun.management.jmxremote.host=localhost

This enables remote anonymous JMX connections on port 5005. You may also consider JVisualVM which is much more please and allows to browse JMX via plugin.

What you are looking for is Catalina -> ThreadPool -> http-bio-8080 -> various interesting metrics.

JMX proxy servlet

Easier method might be to use Tomcat's JMX proxy servlet under: http://localhost:8080/manager/jmxproxy. For instance try this query:

$ curl --user tomcat:tomcat http://localhost:8080/manager/jmxproxy?qry=Catalina:name=%22http-bio-8080%22,type=ThreadPool

A little bit of grepping and scripting and you can easily and remotely monitor your application. Note that tomcat:tomcat is the username/password of user having manager-jmx role in conf/tomcat-users.xml.

You can deploy jolokia.war and then retrieve mbeans values in JSON (without the manager):

http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool?ignoreErrors=true

If you want only some values (currentThreadsBusy, maxThreads, currentThreadCount, connectionCount):

http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool/currentThreadsBusy,maxThreads,currentThreadCount,connectionCount?ignoreErrors=true

{
    request: {
       mbean: "Catalina:name="http-nio-8080",type=ThreadPool",
       attribute: [
          "currentThreadsBusy",
          "maxThreads",
          "currentThreadCount",
          "connectionCount"
       ],
       type: "read"
    },
    value: {
       currentThreadsBusy: 1,
       connectionCount: 4,
       currentThreadCount: 10,
       maxThreads: 200
    },
    timestamp: 1490396960,
    status: 200
}

Note: This example works on Tomcat7 +.

For a more enterprise solution. I have been using New Relic in our production environment.

This provides a graph of the changes to the threadpool over time.

There are cheaper tools out meanwhile: I am using this jar here: https://docs.cyclopsgroup.org/jmxterm You can automate it via shell/batch scripts. I regexed the output and let prometheus poll it for displaying it in grafana.