Nutch web spider, index entire web
Alright, I've been messing around with Nutch and need to know what parameter inside the crawl-urlfilter.txt
file I edit so the spider has no boundaries. In other words I want it to roam around the web outside of a specified domain.
I'm assuming it has to do with this 开发者_开发技巧line, but I don't know how to edit it correctly to do as I want it to:
+^http://([a-z0-9]*\.)*urlz.net/
I'm not framiliar with Nutch but this is just a regular expression.
+^http://([a-z0-9\.])*
Would probably work just fine, or some variation thereof. Its just matching a pattern. The one I just wrote above should match anything starting with http:// and then any number of letters, numbers or dots.
精彩评论