What should i use to crawl many news articles?

I've a project of natural language processing but for that i need to crawl many web articles from some sources like Yahoo news, Google news or blogs...

I'm a java developper (so i'd rather use java tools). I开发者_开发知识库 guess i can parse each source website on my own and extract the articles with HttpClient / XPath but i'm a bit lazy :) is there a way so that i won't have to make a parser per source?

(I'm not only interested by new articles but articles from 2000 to now too)

The hardest part of NLP is getting data you can use. Everything else is just math.

It may be hard to find a large collection of news articles other than on each news source's website because of all the copyright issues involved. If you don't need recent news, your best bet is probably to look at the Linguistic Data Consortium's English Gigaword corpus; if you are at a university, there may already be an existing relationship for you to use the data for free.

If you need to actually crawl and parse websites, for now you'll probably find you have to write specific parsers for the various news websites to make sure you get the right text. However, once more websites start using HTML5, it will be easier to pull out the relevant text through the use of the article tag.

To do the actual crawling, this previous question can point you in some useful directions.