How can you initialize workers with doSMP?
Is there a way to initialize a doSMP cluster similar to clusterEvalQ
and clusterExport
in the snow package? For example:
x <- 1:10
y <- 10:1
z <- rnorm(10)
cl <- makeSOCKcluster(2)
clusterEvalQ(cl, library(quantmod))
clusterExport(cl, list("x","y","z"))
clusterEvalQ(cl, ls())
clusterEvalQ(cl, search())
There is an initEnvir
option to doSMP, but ?doSMP
says
‘initEnvir’ is a function to be executed by each worker before any tasks are executed associated with a foreach. Its purpose is to initialize the execution environment, or the worker in general. It is only executed by a worker if that worker executes at least one task associated with the foreach.
Each worker needs a copy of several large objects in order to run the expression I send to foreach
. Additionally, I need to call foreach
several hundred times, with identical versions of these large objects. It would be inefficient to copy these objects for every call to foreach
.
Even if there isn't a ready-made way to do this, I'd appreciate a kludge.
The first argument to foreach
is ...
which transports the iterated objects for the evaluation of target expression. If you only need part of the object, then it may be more efficient to use iter
to only pass portions of an object.
w <- startWorkers(workerCount = 4)
registerDoSMP(w)
foreach(x=iter(x),y=iter(y),z=iter(z) ) %dopar% (x*y*z)
The objects in the calling environment are still available:
foreach(1:10 ) %dopar% (x*y*z) # Somewhat repetitious #
zed <- 20:1
foreach(x=iter(x) ) %dopar% (x*zed)
Try this:
library(doSMP)
library(foreach)
w <- startWorkers(workerCount = 4)
registerDoSMP(w)
foreach(i = 1:3) %dopar% sqrt(i)
Or you could use doSNOW and
registerDoSNOW(cl)
http://cran.r-project.org/web/packages/doSNOW/doSNOW.pdf
Finally, there is doMC (not for windows)
library(doMC)
registerDoMC()
foreach(i = 1:3) %dopar% sqrt(i)
All these techniques work before the new parallel package in 2.14, which I believe has done some unification of these techniques.
精彩评论