Missing hits on lucene index search

i index one big database overview (just text fields) on which the user must be able to search (below in indexFields method). This search before was done in the database with ILIKE query, but was slow, so now search is done on index. Hovewer, when i compare search results from db query, and results i get with the index search, there is always much less results with search from index. Im not sure if i am making mistake in indexing or in search process. To me all seems to make sense here. Any ideas?

Here is the code. All advices appreciated!

 // INDEXING
StandardAnalyzer analyzer = new StandardAnalyzer(
                Version.LUCENE_CURRENT, stopSet); // stop set is empty
        IndexWriter writer = new IndexWriter(INDEX_DIR, analyzer, true,
                IndexWriter.MaxFieldLength.UNLIMITED);

 开发者_高级运维       indexFields(writer);
        writer.optimize();
        writer.commit();
        writer.close();
        analyzer.close();

private void indexFields(IndexWriter writer) {

    DetachedCriteria criteria = DetachedCriteria
            .forClass(Activit.class);

    int count = 0;
    int max = 50000;
    boolean existMoreToIndex = true;

    List<Activit> result = new ArrayList<Activit>();


    while (existMoreToIndex) {

        try {
            result = activitService.listPaged(count, max);
            if (result.size() < max)
                existMoreToIndex = false;

            if (result.size() == 0)
                return;

            for (Activit ao : result) {
                Document doc = new Document();
                doc.add(new Field("id", String.valueOf(ao.getId()),
                        Field.Store.YES, Field.Index.ANALYZED));
                if(ao.getActivitOwner()!=null)
                    doc.add(new Field("field1", ao.getActivityOwner(),Field.Store.YES, Field.Index.ANALYZED));
                if(ao.getActivitResponsible() != null)
                    doc.add(new Field("field2", ao.getActivityResponsible(), Field.Store.YES,Field.Index.ANALYZED));

                try {
                    writer.addDocument(doc);
                } catch (CorruptIndexException e) {
                    e.printStackTrace();

            }
            count += max;

 //SEARCH
    public List<Activit> searchActivitiesInIndex(String searchCriteria) {
    Set<String> stopSet = new HashSet<String>(); // empty because we do not    want to remove stop words
    Version version = Version.LUCENE_CURRENT;
    String[] fields = {
            "field1", "field2"};
    try {
        File tempFile = new File("C://testindex");
        Directory INDEX_DIR = new SimpleFSDirectory(tempFile);
        Searcher searcher = new IndexSearcher(INDEX_DIR, true);

        QueryParser parser = new MultiFieldQueryParser(version, fields, new StandardAnalyzer(
                version, stopSet));


        Query query = parser.parse(searchCriteria);

        TopDocs topDocs = searcher.search(query, 500);

        ScoreDoc[] hits = topDocs.scoreDocs;


        //here i always get smaller hits lenght

        searcher.close();
    } catch (Exception e) {
        e.printStackTrace();
    }


}

Most likely the analyzer is doing something that you aren't expecting.

Open your index using Luke, you can see what your (analyzed) indexed documents look like, as well as your parsed queries - should let you see what's going wrong.

Also, can you give an example of searchCriteria? And the corresponding SQL query? Without that, it's hard to know if the indexing is done correctly. You may also not need to use MultiFieldQueryParser, which is quite inefficient.