Best way to maintain huge amount of connections in Java
I'm wondering wich is best solution for maintaining huge amount of small TCP connections in multi thread application without locking after some time.
Assume, that we have to visit lot of http web sites (like ~200 000 on different domains, servers etc) in multi thread. Wich classes are the best to do safest connection (I mean most lock-resistance, not multi-threading lock but TCP connection 开发者_运维知识库that will "not react for anything"). Will HttpURLConnection & BufferedReader do the job with setted connection and read timeout ? I saw that when I was using simple solution:
URL url = new URL( xurl );
BufferedReader in = new BufferedReader( new InputStreamReader( url.openStream() ) );
All threads were locked/dead after 2-3 hours.
Is better to have constant threads like 10 running all-time and requesting URL's to take from main thread or better create one thread for each url and then kill it in some way if it will not respond after some time ? (how to kill sub-thread ?)
Well if it is going to be HTTP connection, I really doubt you can cache them. Because keeping the HTTP connection alive is not only at the client side, it requires the server side support too. Most of the time, the server will close the connection after the time out period (which is configured in the server). So, check what is the maximum time out configured at the server side and how long you want to keep the connection cached.
精彩评论