Improving performance of web server

I was recently asked this question in an interview. It was open-ended question on web-server design. Herez the gist of the question.

"Queues quite often become a bottle neck for applications like web servers. What changes will you make to your thread handling classes, thread pool and queues to improve any bottle neck?"

I pointed out measures like 1. Thread pool management 2. Using buffers and packets in queues.

But the interviewer was not quite satisfied. What all design measures should I have mentioned? Obviously its an open-ended question, so answer doesn't have to depend on the underlying technology like Spring, J2EE etc.

In case there i开发者_运维百科s good tutorial on server design, please share it.

I can't resist posting this:

http://www.engineerguy.com/videos/video-lines.htm

The main problem with a queue is that it blocks waiters deeper down the queue. So the design must either be so fast that queues never fill up, or it must take queue fill-up into consideration. Also see the recent things on bufferbloat Jim Gettys have posted:

http://gettys.wordpress.com/category/bufferbloat/

Where buffering of IP packets in modern routers lead to TCP going seriously wrong (and it has effects very indirectly on queuing theory inside web servers as well).

Random answers:

1. Actually measure performance across the system to identify the bottlenecks before randomly making code changes. FTW!

2. Make use of IO Completion ports (Windows), "epoll", or "kevent"(Linux).

3. Consider having different priority queues. For queries that are known to have fast response times, put them in one queue. For queries needing longer and more expensive responses (e.g. long database transactions), put these requests in a different queue. Consider giving more priority to the queue with the fast queries. (Think: express checkout at the grocery store for those with 12 or fewer items).

There is a design pattern called Socket-Wheel that is utilized in advanced web servers for handling queues.

I would have increased the number of queues and threads to lower the latency. (and always have one thread per queue)