Too many open files Exception in JBoss AS 4.2.3-GA

I want to fix this issue, I actually don't have a clear Idea of what is happening when my application runs over a CentOs environment and after some days I start getting the following exceptions:

2011-07-12 21:58:03,598 12155907 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.jav开发者_StackOverflow社区a:408)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
    at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
    at java.lang.Thread.run(Thread.java:662)
2011-07-12 21:58:03,600 12155909 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
    at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
    at java.lang.Thread.run(Thread.java:662)
2011-07-12 21:58:03,600 12155909 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
    at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
    at java.lang.Thread.run(Thread.java:662)
2011-07-12 21:58:03,601 12155910 ERROR [org.jboss.naming.Naming] (JBoss System Threads(1)-2:) Naming accept handler stopping
java.net.SocketException: Too many open files
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
    at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
    at java.lang.Thread.run(Thread.java:662)

The logs start growing quick... I am using some JBOSS QUEUES in my application, as well some JMX connections, I want to know how to detect the problem quickly or if this is caused by other component in the computer. Any suggestions, please I am totally worried about this issue.

Here are some tips on debugging a Too Many Open Files situation. Two unix commands that can be of help are ulimit and lsof

I suggest you man ulimit first to understand that you can alter the maximum number of open files for a process. Just typing ulimit on the commandline will give you the default value for ulimit. For example, for me:

$ ulimit
unlimited

I'm running on a vanilla Ubuntu 11.04 distro, so looks like unlimited is the default. On most of my production boxes, the default is 1024.

Next, lsof -p <pid> which will list all open files for process with id <pid>.

Perhaps you will find that you are either not closing files when thought you were or you are just opening too many files (based on the limit set up ulimit)?

Next step, you are running a Java process so you can get a thread dump of your process while the Too Many Open Files situation is occurring. To get a thread dump, either send the process a kill -3 <pid> or if you have started the Java process in your current shell, you can type CTRL-Break to get a thread dump. It's especially helpful to gather 3 or more thread dumps within, say a minute or two, and from this collection of thread dumps, so the threads that exist in a thread dumps are worth taking a look at.

If none of this helps you, search SO for "too many open files". I did and found this link, which the accepted answer may help you

Java Too Many Open Files

I think you are hitting this issue - AJP connection is left as CLOSE_WAIT