Scraping with multiple IP, in java
Well basically I have a scraping application. It scrapes around n items per minute. currently i have only one IP.
The site i'm scraping allows me 3 connections per IP.
I'm thinking about getting another IP.
so i'll be able to get 6 connections.
in theory i should be able to get n items in 40 seconds, more or less.
currently i'm using java (commons-httpcore) to get the job done.
I'm not sure if this is java question or an OS question.
my machine has IP 1 and IP 2 how 开发者_运维知识库do i connect to, say, www.microsoft.com, using IP 1 and using IP2? how can i specify, which ip i want to use to do a connection?
// requires HttpComponents Client 4.*
DefaultHttpClient httpclient = new DefaultHttpClient();
httpclient.getParams().setParameter(
ConnRoutePNames.LOCAL_ADDRESS,
InetAddress.getByName("10.10.10.10")
);
See: http://hc.apache.org/httpcomponents-client/httpclient/apidocs/index.html
精彩评论