Best way to pre-process text messages using Hadoop

I am using Hadoop to process text messages(SMS). but I am not sure of the best way to pre-process these data so that I can do an efficient search. for example, after preprocess开发者_运维技巧ing the data if someone searches for 'NY' I will be able to display the messages containing the word 'NY'. Is it advisable to write the pre-processed data to an xml file and not to a database.

NOTE: I have around 200K text messages in an .csv file.

The way I import preprocessed data to hdfs is to first import the data (csv file in your case) into a database and then create a table view that fine-tunes it to your needs. Then I import the data into hdfs using Sqoop. More Information on sqoop can be found here

http://www.cloudera.com/blog/2009/06/introducing-sqoop/

for doing a sqoop import from a database take a look at

http://archive.cloudera.com/cdh/3/sqoop/SqoopUserGuide.html#_connecting_to_a_database_server

You probably want to index the text messages, maybe using something like Lucene.

Go for Solr (Especially used for text mining)

1. Powerful full-text search

2. Provides dynamic clustering

3. Provides database integration as well

4. Supports .csv,.xml,word,pdf..

5. Highly scalable