开发者

In Java, there is a collection which I can take an element only after a time?

I'm doing an webcrawler, and I want to not overload the servers with requests, so I will limit the access to the servers by time.

I will have an object/collection that is an list of links that I will visit.

I get the first link and it's from Google(example), visit and do all the work. I only can take another link that is from Google after 20 seconds(example) and while I waiting, I get links from other domains.

Summarizing, I need a collection that allow me to classify the objects inside it and get the objects from it in determined intervals.

There is some similar to this? If not, what collections are re开发者_运维知识库commend to use in the implementation? This collection will be access by various threads.


You can use DelayQueue for this. It's a concurrent collection that only allows items to be taken from it once they have "expired". The items you add must implement Delayed.

You might also just want to use a ScheduledExecutorService. For example, for each domain you were crawling, you could schedule a task that grabs the next link, then re-schedules another task with the next link.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜