开发者

R text mining package: Allowing to incorporate new documents into an existing corpus

I was wondering if there is any chance of R's text mining package having the following feature:

myCorpus <- Corpus(DirSource(<directory-contatining-textfiles>),control=...)
# add docs
myCorpus.addDocs(DirSource(<new-dir>),cont开发者_如何学JAVArol=...)

Ideally I would like to incorporate additional documents into the existing corpus.

Any help is appreciated


You should be able just to use c(,) as in

> library(tm)
> data("acq")
> data("crude")
> together <- c(acq,crude)
> acq
A corpus with 50 text documents
> crude
A corpus with 20 text documents
> together
A corpus with 70 text documents

You can find more in the tm package documentation under tm_combine.


I overcome this issue as well in the context of big data text mining sets. It was not possible to load the entire data set at once.

Here, another option for such big data sets is possible. The approach is to collect a vector of one document corpora inside a loop. After processing all documents like this, it is possible to convert this vector into one huge corpus e.g. to create a DTM on it.

# Vector to collect the corpora:
webCorpusCollection <- c()

# Loop over raw data:
for(i in ...) {

  try({      

    # Convert one document into a corpus:
    webDocument <- Corpus(VectorSource(iconv(webDocuments[i,1], "latin1", "UTF-8")))

    #
    # Do other things e.g. preprocessing...
    #

    # Store this document into the corpus vector:
    webCorpusCollection <- rbind(webCorpusCollection, webDocument)

  })
}

# Collecting done. Create one huge corpus:
webCorpus <- Corpus(VectorSource(unlist(webCorpusCollection[,"content"])))
0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜