开发者

Why is nutch parsing application/x-javascript files?

I configured nutch with the following in my conf/nutch-site.xml

<prope开发者_如何学Pythonrty>
  <name>plugin.includes</name>
  <value>urlfilter-regex|protocol-(http|file)|parse-(text|html|pdf|msword)|in
dex-(basic|anchor|more)|query-(basic|site|url)|response-(json|xml)|summary-basic
|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
  <description>Regular expression naming plugin directory names to
  include.  Any plugin not matching this expression is excluded.
  In any case you need at least include the nutch-extensionpoints plugin. By
  default Nutch includes crawling just HTML and plain text via HTTP,
  and basic indexing and search plugins. In order to use HTTPS please enable 
  protocol-httpclient, but be aware of possible intermittent problems with the 
  underlying commons-httpclient library.
  </description>
</property>

Note the list of parsers - only text, html, pdf and msword. But for some strange reason I just discovered some application/x-javascript files in my index. Why would that be? Is it using what's in the plugins directory and disregarding my plugin.includes?


I use Nutch 1.1 (not trunk) to parse rss feeds. I use the parse-rss plugin. Feeds are only parsed if I activate the plugin. If not, they are ignored. So to answer your question, yes, Nutch should only be using the plugins defined in plugin.includes.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜