开发者

Nutch web spider, index entire web

Alright, I've been messing around with Nutch and need to know what parameter inside the crawl-urlfilter.txt file I edit so the spider has no boundaries. In other words I want it to roam around the web outside of a specified domain.

I'm assuming it has to do with this 开发者_开发技巧line, but I don't know how to edit it correctly to do as I want it to:

+^http://([a-z0-9]*\.)*urlz.net/


I'm not framiliar with Nutch but this is just a regular expression.

+^http://([a-z0-9\.])*

Would probably work just fine, or some variation thereof. Its just matching a pattern. The one I just wrote above should match anything starting with http:// and then any number of letters, numbers or dots.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜