Java socketserver: How to handle many incoming connections?

I am writing a simple multithreaded socketserver and I am wondering how best to handle incoming connections:

1. create a new thread for each new connection. The number of concurrent threads would be limited and waiting connections limited by specifying a backlog

2. add all incoming connections into a queue and have a pool of worker threads that process the queue

I am inclined to go for option 2 because I really don't want to refuse any connections, even under high loads, but I am wondering if there are any considerations I should be awa开发者_JAVA技巧re of with accepting effectively unlimited connections?

With unlimited connections you potentially can create a huge number of threads. That means a lot of processing is required, plus each thread will consume a fixed amount of memory by default merely for the heap (I think the figure is 512kB per thread, but that may be platform dependent).

By pooling a fixed number of threads and accepting a limited number of clients you will ensure that some of your clients will be serviced in a reasonable time period, and your server won't collapse from overloading.

You may want to check out this article on building servers using NIO or perhaps check out frameworks such as Apache Mina.

I really don't want to refuse any connections

Actually you probably do. When you're overloaded, you want to retain enough capacity to get rid of the current load before you accept any more. Slowing everdybody to a halt isn't any more acceptable than refusing connections.

Queueing theory says that the sweet spot is about 70% utilization. If your server is going to have a steady load higher than that get faster hardware until it doesn't.

Having said that, if you're expecting hundreds of thousands of connections I would use a thread pool or NIO. If you're only expecting thousands, a thread per connection is much the easiest way to go.

If you accept unlimited connections, and your processing isn't fast enough to cope with the stream of incoming connections, your queue will fill up - until it reaches a point, where it takes so long to process previous requests, that most requests won't even be interested in the answer anymore, as soon as you get to them.

1 Thread per connection does not scale. Read about it at The C10K problem. It is a long read, but very famous.

I would advise to learn about NIO or rather use the impressive netty framework which handles all the heavy-lifting(NIO) for you.

I really don't want to refuse any connections, even under high loads

Your option 2 is the only way to go in my opinion. You should have one pinned NIO selector thread per each 5-10k connections. But nothing can delay this critical thread, so you use a DEMUX to distribute work among a fixed number of pinned threads and not to a thread pool. And a MUX to collect work and respond back to clients. As EJP said, if your worker threads are lagging, eventually you will have to drop connections unless you start dumping messages to disk to make your queue as large as possible. That way y won't be dropping any connection, even under high loads. You can check this article which explain this in detail and the diagram below:

Disclaimer: I am one of the developers of CoralQueue