开发者

Improve Efficiency for This Text Processing Code

I am writing a program that counts the number of words in a text file which is already in lowercase and separated by spaces. I want to use a dictionary and only count the word IF it's within the dictionary. The problem is the dictionary is quite large (~100,000 words) and each text document has also ~50,000 words. As such, the codes that I wrote below gets very slow (takes about 15 sec to process one document on a quad i7 machine). I'm wondering if there's something wrong with my coding and if the efficiency of the program can be improved. Thanks so much for your help. Code below:

public static string WordCount(string countInput)
        {
            string[] keywords = ReadDic(); /* read dictionary txt file*/

            /*then reads the main text file开发者_JAVA技巧*/
            Dictionary<string, int> dict = ReadFile(countInput).Split(' ')
                .Select(c => c)
                .Where(c => keywords.Contains(c))
                .GroupBy(c => c)
                .Select(g => new { word = g.Key, count = g.Count() })
                .OrderBy(g => g.word)
                .ToDictionary(d => d.word, d => d.count);

            int s = dict.Sum(e => e.Value);
            string k = s.ToString();
            return k;

        } 


You can vastly improve performance by reading the text file one line at a time instead of building an enormous string.

You can call

File.ReadLines(path).SelectMany(s => s.Split(' '))

Do not call ReadAllLines; it will need to build an enormous array.


Your first Select call is utterly useless.


Your Contains call will loop through the entire dictionary for each word in the file.
Thus, the Where call is an O(n2) operation.

Change keywords to a HashSet<string>.
Since HashSets can be searched in constant time, the Where call will become an O(n) operation, which is much better.


Your second Select call can be combined with the GroupBy, which will cut a large number of object allocations:

 .GroupBy(c => c, (word, set) => new { word, count = set.Count() })

Dictionaries are intrisically unordered, so your OrderBy call is a useless waste of time.


Since you're running quad-core, you could at least throw in an AsParallel() in there.


As I can see all your code can be replaced with

return ReadFile(countInput).Split(' ').Count(c => keywords.Contains(c));

And, as SLaks said, HashSet - would improve performance.
One more improvement: if you call this code in loop, the you shouldn't read ReadDic() on each iteration - load it once and pass as a parameter.


Try changing string[] keywords to a HashSet<string> keywords. Your call to "Contains" is essentially a loop, which is going to be vastly slower than a lookup by hash key.

If you want to get REALLY fancy, you could make use of multiple threads by using some PLINQ, but I would make sure you've optimized your single thread performance before going that route.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜