How to use the source() function in a parallized loop (doSMP)?
I am trying to speed up my code with a 'foreach' loop using the doSMP package.
Here is a simplified version of my issue: I am running a file called main.R
file: main.R:
require(doSMP)
dropbox_path = "/home/ruser/Dropbox"
workers <- startW开发者_开发问答orkers(4)
registerDoSMP(workers)
foreach(jj=1:4 ) %dopar% source("test.R")
stopWorkers(workers)
file: test.R:
message(dropbox_path)
This returns the following error: "Error in source("test.R") : task 1 failed - "object 'dropbox_path' not found"
If I modify main.R to be :
require(doSMP)
dropbox_path = "/home/ruser/Dropbox"
workers <- startWorkers(4)
registerDoSMP(workers)
foreach(jj=1:4 ) %dopar% message(dropbox_path)
stopWorkers(workers)
It then works very well. It also used to work well with sequencial code ('for' instead of 'foreach').
So R child instances can access the dropbox_path variable, but not when it is parsed through the source function. I tried to play around with the source() function arguments 'local' and 'chdir' with no sucess.
Would you know a way for the code to work? I would like to keep using the source() function.
I apologize in advance that my little workaround doesn't use your tools, but here's how I would do it. I use package snowfall
because I can easily extend my apply functions to work on multiple cores. The code is not tested because all of my cores are currently occupied. Should work, though.
tiny_script.R
contents:
date()
R code to fire up multiple cores:
library(snowfall)
sfInit(parallel = TRUE, cpus = 4, type = "SOCK") #power up
my.list <- vector("list", 10)
sfLapply(x = my.list, fun = function(x) source("./Odpad/tiny_script.R"))
Running on a single core using lapply
only:
> lapply(X = my.list, FUN = function(x) source("./Odpad/tiny_script.R")) #notice the difference in argument names between `lapply` and `sfLapply`.
[[1]]
[[1]]$value
[1] "Wed Jun 22 13:02:11 2011"
[[1]]$visible
[1] TRUE
[[2]]
[[2]]$value
[1] "Wed Jun 22 13:02:11 2011"
精彩评论