What is the simplest way of parallelization over a cluster with SSH and NFS?

I have a lot of trivially parallelizable computations and a lot (100s) of cores distributed overs SSH + NFS network. What is the simplest way of parallelization. The problem is that I don't know how long each task will take so I need some kind o开发者_运维知识库f queue. Is there something that is very easy to use?

Mincemeat.py is a lightweight Python implementation of MapReduce designed just for this purpose. There's not an easy way to launch all of the processes (unless you have something like Condor on the cluster), but otherwise it should fit your needs.

You may also have a look at mpssh & autossh.

http://www.mynitor.com/2010/08/16/top-50-ssh-helper-tools-omg/

I have a lot of trivially parallelizable computations and a lot (100s) of cores distributed overs SSH + NFS network. What is the simplest way of parallelization. The problem is that I don't know how long each task will take so I need some kind of queue. Is there something that is very easy to use?

I'm answering late, but I went on a similar search. I've been using this for a while and I'm pretty convinced that orgalorg is the answer.

Linux and macOS are supported.

Example below...