R text mining package: Allowing to incorporate new documents into an existing corpus
I was wondering if there is any chance of R's text mining package having the following feature:
myCorpus <- Corpus(DirSource(<directory-contatining-textfiles>),control=...)
# add docs
myCorpus.addDocs(DirSource(<new-dir>),cont开发者_如何学JAVArol=...)
Ideally I would like to incorporate additional documents into the existing corpus.
Any help is appreciated
You should be able just to use c(,)
as in
> library(tm)
> data("acq")
> data("crude")
> together <- c(acq,crude)
> acq
A corpus with 50 text documents
> crude
A corpus with 20 text documents
> together
A corpus with 70 text documents
You can find more in the tm package documentation under tm_combine
.
I overcome this issue as well in the context of big data text mining sets. It was not possible to load the entire data set at once.
Here, another option for such big data sets is possible. The approach is to collect a vector of one document corpora inside a loop. After processing all documents like this, it is possible to convert this vector into one huge corpus e.g. to create a DTM on it.
# Vector to collect the corpora:
webCorpusCollection <- c()
# Loop over raw data:
for(i in ...) {
try({
# Convert one document into a corpus:
webDocument <- Corpus(VectorSource(iconv(webDocuments[i,1], "latin1", "UTF-8")))
#
# Do other things e.g. preprocessing...
#
# Store this document into the corpus vector:
webCorpusCollection <- rbind(webCorpusCollection, webDocument)
})
}
# Collecting done. Create one huge corpus:
webCorpus <- Corpus(VectorSource(unlist(webCorpusCollection[,"content"])))
精彩评论