How Lucene scores results in a RegexQuery?

I can see how two values, wh开发者_JAVA技巧en doing a regular/fuzzy full text search, can be compared to determine which one is "better" (i.e. one value contains more keywords than the other, one contains less non-keywords than the other).

However, how Lucene computes the score when doing regex queries using RegexQuery? It is a boolean query - a field's value is either compatible with the regex or not. Lucene can't take keywords from my regex query and do its usual magic...

There are two passes. In the first, it generates a list of all terms which match the regex. In the second, it finds all documents with terms matching that regex.

The major code you want to look at is in MultiTermQuery:

public Query rewrite(IndexReader reader) throws IOException {
  FilteredTermEnum enumerator = getEnum(reader);
  BooleanQuery query = new BooleanQuery();
  try {
    do {
      Term t = enumerator.term();
      if (t != null) {
        TermQuery tq = new TermQuery(t);      // found a match
        tq.setBoost(getBoost() * enumerator.difference()); // set the boost
        query.add(tq, false, false);          // add to query
      }
    } while (enumerator.next());
  } finally {
    enumerator.close();
  }
  return query;
}

Two things:

1. The boolean query is instantiated with coord on. So the standard coord scoring applies (i.e. the more terms you get, the better).
2. The boost of the term query is given by enumerator.difference(). However, as of 3.0.1 this just returns 1:

   
   @Override
   public final float difference() {
   // TODO: adjust difference based on distance of searchTerm.text() and term().text()
   return 1.0f;
   }
   

   So at some point this will return the distance (probably levenstein) between the terms. But for now it does nothing.

This is just a wild guess, but one possible metric could be the number of backtracking steps the regex engine needs to take to match your search strings.

Of course, these values also depend mightily on the quality of your regex, but when comparing several matches, the one that was "easier to match" could be considered a better match than the one that the regex engine had to go through contortions for.