Question about parallel loop in MPI

hey there, I have a short question about openmpi in fortran: i have a code like this:

I) definitions of vars & linear code, setting up some vars for later usage
II) a while loop which works like that in pseudocode:

nr=1
while(true)
{
  filename='result'//nr//'.bin' (nr converted to string)
  if(!file_exists(filename))
    goto 100

  // file exists... so do something with it
  // calculations, read/write...
  nr=nr+1
}
100 continue
III) some more linear code...

Now I want to make this a parallel computation with openmpi. The linear code from I) and III) should only be computed once and the while-loops should be run on several processors... How to best realize it? my problem is how the while-loop works: E.g. when processor 1 computes result1.bin, how to directly tell processor 2 to compute result2.bin? and how would it work if there are 30 files and I use

mpirun -n 10 my_program

? how does MPI "know" that after finishing computing one file, there are more files "waiting" to be processed: as soon as one processor has ended processing one file, THIS procesor should directly start over processing the next file in the queue..

thanks so far!

EDIT:

Hey there, it's me again... I wanted to give OpenMP a try too, so I used a chunk of your code which reads the existing files and afterwards loops them (and processes them):

nfiles = 0
do
  write(filename,FMT='(A,I0,A)'), prefix, nfiles+1, suffix
  inquire(file=trim(filename),exist=exists)
  if (not(exists)) exit
    nfiles = nfiles + 1
enddo

now I tried the following code:

call omp_set_num_threads(2)
!$OMP PARALLEL
!$OMP DO 
do i=startnum, endnum
  write(filename,FMT='(A,I0,A)'), prefix, i, suffix
  ...CODE DIRECTLY HERE TO PROCESS THE FILE...
enddo
!$OMP END DO
!$OMP END PARALLEL

But it gives me always errors like that: "It is illegal to branch out of a DO loop a开发者_运维知识库ssociated with an Open MP DO or PARALLEL DO directive."

Always about the codelines with this sort of code:

read (F_RESULT,*,ERR=1) variable

Where F_RESULT is a filehandle...What could be wrong about it? variable is defined outside of the loop block, and I already tried to set OpenMP directive to

private(variable) 

so that each thread has his own copy, but that didn't work out! Thanks so far for your help!

Probably the most sensible way to do this is to have one of the processes count the total number of files beforehand, broadcast that, and then have everyone do "their" files:

program processfiles
    use mpi
    implicit none

    integer :: rank, comsize, ierr
    integer :: nfiles
    character(len=6) :: prefix="result"
    character(len=4) :: suffix=".bin"
    character(len=50) :: filename
    integer :: i
    integer :: locnumfiles, startnum, endnum
    logical :: exists

    call MPI_Init(ierr)
    call MPI_Comm_size(MPI_COMM_WORLD, comsize, ierr)
    call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierr)

    ! rank zero finds number of files
    if (rank == 0) then
       nfiles = 0
       do
           write(filename,FMT='(A,I0,A)'), prefix, nfiles+1, suffix
           inquire(file=trim(filename),exist=exists)
           if (not(exists)) exit
           nfiles = nfiles + 1
       enddo
    endif
    ! make sure everyone knows
    call MPI_Bcast(nfiles, 1, MPI_INTEGER, 0, MPI_COMM_WORLD, ierr)

    if (nfiles /= 0) then
        ! calculate who gets what file
        locnumfiles = nfiles/comsize
        if (locnumfiles * comsize /= nfiles) locnumfiles = locnumfiles + 1
        startnum = locnumfiles * rank + 1
        endnum = startnum + locnumfiles - 1
        if (rank == comsize-1) endnum = nfiles
        do i=startnum, endnum
           write(filename,FMT='(A,I0,A)'), prefix, i, suffix
           call processfile(rank,filename)
        enddo
    else
        if (rank == 0) then
            print *,'No files found; exiting.'
        endif
    endif
    call MPI_Finalize(ierr)

    contains
        subroutine processfile(rank,filename)
            implicit none
            integer, intent(in) :: rank
            character(len=*), intent(in) :: filename
            integer :: unitno
            open(newunit=unitno, file=trim(filename))
            print '(I4,A,A)',rank,': Processing file ', filename
            close(unitno)
        end subroutine processfile
end program processfiles

And then a simple test:

$ seq 1 33 | xargs -I num touch "result"num".bin"
$ mpirun -np 2 ./processfiles

   0: Processing file result1.bin                                       
   0: Processing file result2.bin                                       
   0: Processing file result3.bin                                       
   0: Processing file result4.bin                                       
   0: Processing file result5.bin                                       
   0: Processing file result6.bin                                       
   1: Processing file result18.bin                                      
   0: Processing file result7.bin                                       
   0: Processing file result8.bin                                       
   1: Processing file result19.bin                                      
   0: Processing file result9.bin                                       
   1: Processing file result20.bin                                      
   0: Processing file result10.bin                                      
   1: Processing file result21.bin                                      
   1: Processing file result22.bin                                      
   0: Processing file result11.bin                                      
   1: Processing file result23.bin                                      
   0: Processing file result12.bin                                      
   1: Processing file result24.bin                                      
   1: Processing file result25.bin                                      
   0: Processing file result13.bin                                      
   0: Processing file result14.bin                                      
   1: Processing file result26.bin                                      
   1: Processing file result27.bin                                      
   0: Processing file result15.bin                                      
   0: Processing file result16.bin                                      
   1: Processing file result28.bin                                      
   1: Processing file result29.bin                                      
   1: Processing file result30.bin                                      
   0: Processing file result17.bin                                      
   1: Processing file result31.bin                                      
   1: Processing file result32.bin                                      
   1: Processing file result33.bin  

Updated to add the supplementary OpenMP question:

So that first loop is where the number of files is calculated, before the parallel processing of the files starts. That counting of the files needs to be done before the parallel processing of files can happen, because otherwise it's not possible to divide up the work between the processors; you need to know how many "work units" there are going to be before partitioning the work. (This isn't absolutely the only way to do things, but it's the most straightforward).

Similarly, OMP DO loops require quite structured loops -- there needs to be a simple loop like do i=1,n that can then be easily broken up between threads. n doesn't need to be compiled in, and the increment doesn't even need to be one, but it must be the sort of thing that can be known for sure before the loop is actually executed. So, for instance, you can't exit out of the loop because of some external cause (like the non-presence of a file.)

So what you'd want to do with OpenMP is to do the same file counting, and leave that alone, but then in the processing loop, use a parallel do construct. So, after stripping out the MPI stuff, you'd have something that looks like:

    do
        write(filename,FMT='(A,I0,A)'), prefix, nfiles+1, suffix
        inquire(file=trim(filename),exist=exists)
        if (.not.exists) exit
        nfiles = nfiles + 1
    enddo

    if (nfiles /= 0) then
        !$OMP PARALLEL SHARED(nfiles,prefix,suffix) PRIVATE(i,thread,filename)
        thread = omp_get_thread_num()
        !$OMP DO 
        do i=1, nfiles
           write(filename,FMT='(A,I0,A)'), prefix, i, suffix
           call processfile(thread,filename)
        enddo
        !$OMP END DO
        !$OMP END PARALLEL 
    else
        print *,'No files found; exiting.'
    endif

but everything else would be the same. And again, if you want to process the files "inline" (eg, not in a sburoutine), you'd put the file-processing code in where the 'call processfile()' line is.