word distribution problem

I have a big file of words ~100 Gb and have limited memory 4Gb. I need to calculate word distribution from this file. Now one option is to divide it into chunks and sort each chunk and then merge to calculate word distribution. Is there any other way it can be done faster? One idea is to sample but not sure how to implement it to return c开发者_运维知识库lose to correct solution.

Thanks

You can build a Trie structure where each leaf (and some nodes) will contain the current count. As words will intersect with each other 4GB should be enough to process 100 GB of data.

Naively I would just build up a hash table until it hits a certain limit in memory, then sort it in memory and write this out. Finally, you can do n-way merging of each chunk. At most you will have 100/4 chunks or so, but probably many fewer provided some words are more common than others (and how they cluster).

Another option is to use a trie which was built for this kind of thing. Each character in the string becomes a branch in a 256-way tree and at the leaf you have the counter. Look up the data structure on the web.

If you can pardon the pun, "trie" this:

public class Trie : Dictionary<char, Trie>
{
    public int Frequency { get; set; }

    public void Add(string word)
    {
        this.Add(word.ToCharArray());
    }

    private void Add(char[] chars)
    {
        if (chars == null || chars.Length == 0)
        {
            throw new System.ArgumentException();
        }

        var first = chars[0];
        if (!this.ContainsKey(first))
        {
            this.Add(first, new Trie());
        }

        if (chars.Length == 1)
        {
            this[first].Frequency += 1;
        }
        else
        {
            this[first].Add(chars.Skip(1).ToArray());
        }
    }

    public int GetFrequency(string word)
    {
        return this.GetFrequency(word.ToCharArray());
    }

    private int GetFrequency(char[] chars)
    {
        if (chars == null || chars.Length == 0)
        {
            throw new System.ArgumentException();
        }

        var first = chars[0];
        if (!this.ContainsKey(first))
        {
            return 0;
        }

        if (chars.Length == 1)
        {
            return this[first].Frequency;
        }
        else
        {
            return this[first].GetFrequency(chars.Skip(1).ToArray());
        }
    }
}

Then you can call code like this:

var t = new Trie();

t.Add("Apple");
t.Add("Banana");
t.Add("Cherry");
t.Add("Banana");

var a = t.GetFrequency("Apple"); // == 1
var b = t.GetFrequency("Banana"); // == 2
var c = t.GetFrequency("Cherry"); // == 1

You should be able to add code to traverse the trie and return a flat list of words and their frequencies.

If you find that this too still blows your memory limit then might I suggest that you "divide and conquer". Maybe scan the source data for all the first characters and then run the trie separately against each and then concatenate the results after all of the runs.

do you know how many different words you have? if not a lot (i.e. hundred thousand) then you can stream the input, determine words and use a hash table to keep the counts. after input is done just traverse the result.

Just use a DBM file. It’s a hash on disk. If you use the more recent versions, you can use a B+Tree to get in-order traversal.

Why not use any relational DB? The procedure would be as simple as:

1. Create a table with the word and count.
2. Create index on word. Some databases have word index (f.e. Progress).
3. Do SELECT on this table with the word.
4. If word exists then increase counter.
5. Otherwise - add it to the table.

If you are using python, you can check the built-in iter function. It will read line by line from your file and will not cause memory problems. You should not "return" the value but "yield" it. Here is a sample that I used to read a file and get the vector values.

def __iter__(self):  
     for line in open(self.temp_file_name):
         yield self.dictionary.doc2bow(line.lower().split())