Best way to keep index real time?

I have a Solr/Lucene index file of approximately 700 Gb. The documents that I need to index are being read in real-time, roughly 1000 docs every 30 minutes are submitted and need to be indexed. In my scenario 开发者_Go百科a script is run every 30 mins that indexes the documents that are not yet indexed, since it is a requirement that new documents should be searchable as soon as possible, but this process slow down the searching.

Is this the best way i can index latest documents or there is some other better way!

First, remember that Solr is not a real-time search engine (yet). There is still work to be done.

You can use a master/slave setup, where the indexation are done on the master and the search on the slave. With this, indexation does not affect search performance. After the commit is done on the master, force the slave to fetch the latest index from the master. While the new index is being replicated on the slave, it is still processing queries with the previous index.

Also, check you cache warming settings. Remember that this might slow down the searches if those settings are too aggressive. Also check the queries launched on the new searcher event.

You can do this with Lucene easily. Split the indexes in multiple parts (or to be precise, while building indexes, create "smaller" parts.) Create searcher for each of the part and store a reference to them. You can create a MultiSearcher on top of these individual parts.

Now, there will be only one index that will get the new documents. At regular intervals, add documents to this index, commit and re-open this searcher.

After the last index is updated, you can create a new multi-searcher again, using the previously opened searchers.

Thus, at any point, you will be re-opening only one searcher and that will be quite fast.

Check http://code.google.com/p/zoie/ wrapper around Lucene to make it real time - code donated from Linkedin.

^^i do this, with normal lucene, non solr, and it works really nice. however not sure if there is a solr way to do that at the moment. twitter recently went with lucene for searching and has effectively real time searching by just writing to their index at any update. their index resides completely in memory, so updating/reading the index is of no consequence and happens instantly, a lucene index can always be read while being written to, as long as there is only one writer at a time.

Check out this wiki page