configuring nutch regex-normalize.xml
I am using the Java-based Nutch web-search software. In order to prevent duplicate (url) results from being returned in my search query results, I am trying to remove (a.k.a. normalize) the expressions of 'jsessionid' from the urls being indexed when running the Nutch crawler to index my intranet. However my modifications to $NUTCH_HOME/conf/regex-normalize.xml (prior to running my crawl) do not seem to be having any effect.
How can I ensure that my regex-normalize.xml configuration is being engaged for my crawl? and,
What regular expression will successfully remove/normalize expressions of 'jsessionid' from the url during the crawl/indexing?
The following is the contents of my current regex-normalize.xml:
<?xml version="1.0"?>
<regex-normalize>
<regex>
<pattern>(.*);jsessionid=(.*)$</pattern>
<substitution>$1</substitution>
</regex>
<regex>
<pattern>(.*);jsessionid=(.*)(\&|\&amp;)</pattern>
<substitution>$1$3</substitution>
</regex>
<regex>
<开发者_如何学JAVApattern>;jsessionid=(.*)</pattern>
<substitution></substitution>
</regex>
</regex-normalize>
Here is the command that I am issuing to run my (test) 'crawl':
bin/nutch crawl urls -dir /tmp/test/crawl_test -depth 3 -topN 500
What version of Nutch are you using? I'm not familiar with Nutch but the default download of Nutch 1.0 already contains a rule in regex-normalize.xml which seems to handle this problem.
<!-- removes session ids from urls (such as jsessionid and PHPSESSID) -->
<regex>
<pattern>([;_]?((?i)l|j|bv_)?((?i)sid|phpsessid|sessionid)=.*?)(\?|&|#|$)</pattern>
<substitution>$4</substitution>
</regex>
Btw. regex-urlfilter.txt seems to contain something of relevance too
# skip URLs containing certain characters as probable queries, etc.
-[?*!@=]
Then there are some settings in nutch-default.xml which you might want to check out
urlnormalizer.order
urlnormalizer.regex.file
plugin.includes
If that all doesn't help maybe this does: How can I force fetcher to use custom nutch-config?
精彩评论