QSUB a process for every file in a directory?

I've been using

qsub -t 1-90000 do_stuff.sh

to submit my tasks on a Sun GridEngine cluster, but now find myself with data sets (super large ones, too) which are not so conveniently named. What's the best way to go about this? I could try to rename them all, but the names contain information which needs to be preserved, and this obviously introduc开发者_C百科es a host of problems. I could just preprocess everything into jsons, but if there's a way to just qsub -all_contents_of_directory, that would be ideal.

Am I SOL? Should I just go to the directory in question and find . -exec 'qsub setupscript.sh {}'?

Use another script to submit the job - here's an example I used where I want the directory name in the job name. "run_openfoam" is the pbs script in the particular directory.

#!/bin/bash
cd $1
qsub -N $1 run_openfoam

You can adapt this script to suit your job and then run it through a loop on the command line. So rather than submitting a job array, you submit a job for each dir name passed as the first parapmeter to this script.

I tend to use Makefiles to automate this stuff:

INPUTFILES=$(wildcard *.in)
OUTPUTFILES=$(patsubst %.in,%.out,$(INPUTFILES))

all : $(OUTPUTFILES)

%.out : %.in
        @echo "mycommand here < $< > $@" | qsub 

Then type 'make', and all files will be submitted to qsub. Of course, this will submit everything all at once, which may do unfortunate things to your compute cluster and your sysadmin's blood pressure.

If you remove the "| qsub", the output of make is a list of commands to run. Feed that list into one or more qsub commands, and you'll get an increase in efficiency and a reduction in qsub jobs. I've been using GNU parallel for that, but it needs a qsub that blocks until the job is done. I wrote a wrapper that does that, but it calls qstat a lot, which means a lot of hitting on the system. I should modify it somehow, but there aren't a lot of computationally 'good' options here.

I cannot understand "-t 1-90000" in your qsub command. My searching of qsub manual doesn't show such "-t" option.

Create a file with a list of the datasets in it find . -print >~/list_of_datasets Script:

#!/bin/bash
exec ~/setupscript.sh $(sed -n -e "${SGE_TASK_ID}p" <~/list_of_datasets)

qsub -t 1-$(wc -l ~/list_of_datasets) job_script