开发者

anemone ignore url links including a certain phrase

I am running a web scraper开发者_如何学运维 with anemone on ruby and I am giving my server some problems when it visits pages that require a logon.

The pages all have a phrase, say, "account" in the url and I want the program to completely ignore and not go to any link with a destination containing this string.

How can I do this?


Anemone has a skip_links_like method:

skip_links_like(*patterns)
Add one ore more Regex patterns for URLs which should not be followed

So adding something like

skip_links_like /\/account\//

should take care of it:

Anemone.crawl("somesite.co.uk", :depth_limit => 1) do |anemone|
    anemone.skip_links_like /\/account\//
    #...
end
0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜