开发者

Discovering "templates" in a given text?

If I have significant amounts of text and am trying to discover templates that occur most frequently, I was thinking of solving it using the N-Gram approach and开发者_Python百科 in fact it was suggested as a solution in this question as well but my requirement is slightly different. Just to clarify, I have some text like this:

I wake up every day morning and read the newspaper and then go to work
I wake up every day morning and eat my breakfast and then go to work
I am not sure that this is the solution but I will try
I am not sure that this is the answer but I will try
I am not feeling well today but I will get the work done and deliver it tomorrow
I was not feeling well yesterday but I will get the work done and let you know by tomorrow

and am trying to extract "templates" like this:

I wake up every day morning and ... and then go to work
I am not sure that this is the ... but I will try
I ... not feeling well ... but I will get the work done and ... tomorrow

I am looking for an approach that can scale to million of lines of text so I was just wondering if I can adapt the same N-gram approach to solve this problem or are there any alternatives?


Millions of lines of text isn't a really big number :)

What you're looking for is at least similar to collocation finding. You could try to compute pointwise mutual information on n-grams. See Manning & Schütze (1999) for this and other approaches to the problem.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜