Improve Efficiency for This Text Processing Code
I am writing a program that counts the number of words in a text file which is already in lowercase and separated by spaces. I want to use a dictionary and only count the word IF it's within the dictionary. The problem is the dictionary is quite large (~100,000 words) and each text document has also ~50,000 words. As such, the codes that I wrote below gets very slow (takes about 15 sec to process one document on a quad i7 machine). I'm wondering if there's something wrong with my coding and if the efficiency of the program can be improved. Thanks so much for your help. Code below:
public static string WordCount(string countInput)
{
string[] keywords = ReadDic(); /* read dictionary txt file*/
/*then reads the main text file开发者_JAVA技巧*/
Dictionary<string, int> dict = ReadFile(countInput).Split(' ')
.Select(c => c)
.Where(c => keywords.Contains(c))
.GroupBy(c => c)
.Select(g => new { word = g.Key, count = g.Count() })
.OrderBy(g => g.word)
.ToDictionary(d => d.word, d => d.count);
int s = dict.Sum(e => e.Value);
string k = s.ToString();
return k;
}
You can vastly improve performance by reading the text file one line at a time instead of building an enormous string.
You can call
File.ReadLines(path).SelectMany(s => s.Split(' '))
Do not call ReadAllLines
; it will need to build an enormous array.
Your first Select
call is utterly useless.
Your Contains
call will loop through the entire dictionary for each word in the file.
Thus, the Where
call is an O(n2) operation.
Change keywords
to a HashSet<string>
.
Since HashSets can be searched in constant time, the Where
call will become an O(n) operation, which is much better.
Your second Select
call can be combined with the GroupBy
, which will cut a large number of object allocations:
.GroupBy(c => c, (word, set) => new { word, count = set.Count() })
Dictionaries are intrisically unordered, so your OrderBy
call is a useless waste of time.
Since you're running quad-core, you could at least throw in an AsParallel()
in there.
As I can see all your code can be replaced with
return ReadFile(countInput).Split(' ').Count(c => keywords.Contains(c));
And, as SLaks said, HashSet - would improve performance.
One more improvement: if you call this code in loop, the you shouldn't read ReadDic()
on each iteration - load it once and pass as a parameter.
Try changing string[] keywords
to a HashSet<string> keywords
. Your call to "Contains" is essentially a loop, which is going to be vastly slower than a lookup by hash key.
If you want to get REALLY fancy, you could make use of multiple threads by using some PLINQ, but I would make sure you've optimized your single thread performance before going that route.
精彩评论