What are indexes in Lucene?

What are the indexes in Lucene and how it works?

I have gone through some articles on net and google but I could not understand the concept of the index, documents etc fully.

Please help if anyone can explain in s开发者_JAVA百科imple terms the term index and the indexing.

Thanks !

Say you have a bunch of information you would like to make searchable. For example, some HTML files, some PDFs, and some information stored in a database. When a user does a search, you could write a search engine that trawls through this information and return results that match. However, this is typically way too slow for large sets of data.

So in advance of running our application, we create an index of the information that needs to be searchable. The index contains a summary of each piece of information we would like to include in the search. In Lucene, the summary for an information piece is called a document. A document contains a number of fields.

When creating the index you decide which fields to include based on what you would like to be searchable. For example, you may include a title, an id, category string and so forth. Once the fields are defined you create a document in the index for each information item (html, pdf, database entries etc). This process is called indexing.

The search engine can now use the index to search for things. The index is highly optimized for the typical searches that we do. You can search for information in specific fields and do boolean logic. You can search for precise matches or fuzzy ones. And the search engine will weigh/score your documents in the index, returning the most relevant first.

Hope that helps at a high level.

Lucene creates an inverted full-text index, it splits the documents into words, builds an index for each word.

For Instance:
Document 1: "Apache Lucene Java"
Document 2: "Java Library"

Inverted Index:
Tokens       Document Location
apache      1
Library     2
Java        1, 2
Lucene      1

Lets expand is further, now lets consider Document with two Fields. Body and Title.

Document doc = new Document()
doc.add(new Field("body", "This is my Test document", Field.Store.YES, Field.Index.TOKENIZED)
doc.add(new Field("title", "Test document", Field.Store.YES, Field.Index.UNTOKENIZED)

You have the flexibility to tokenize or not tokenize a Field.

Luncene has various analyzer, using the StandardAnalyzer

Analyzer analyzer = new StandardAnalyzer()

above document would be tokenized "my", "Test", "document", "test document"