开发者

paralle::forkmanager perl module

I have a large array about 10000 data and want to process it with parallel::forkmanager

how can i proccess the data on chunck of 1000 i have the below code :

my $MAX_PROCESSES = 10;

 my  $pm = new Parallel::ForkManager($MAX_PROCESSES);

    for (<>) {
       my $pid = $pm->start and next; 
      #  here i want to process my data on chunks of 1000 with 10 parallel::formanagaer
       $pm->finish; 
         }

How can I custo开发者_如何学JAVAmize my code to do it ?


As the docs show, the data is passed from the parent to the child, so you want something that takes the following form:

for (;;) {
   ... get a chunk ...
   my $pid = $pm->start and next; 
   ... process chunk ...
   $pm->finish; 
}

So

use constant CHUNK_SIZE => 1000;

CHUNK:    
for (my $eof = 0; !$eof;) {
   my @chunk;
   while (@chunk < CHUNK_SIZE) {
      my $line = <>;
      if (!$line) {
         if (@chunk) {
            $eof = 1;  # Can't rely on a handle returning EOF twice.
            last;      #   so we have to make a note of it.
         } else {
            last CHUNK;
         }
      }

      push @chunk, $line;
   }

   my $pid = $pm->start and next; 
   ... process chunk ...
   $pm->finish; 
}


my $max_procs = 4;
my $pm = new Parallel::ForkManager($max_procs);

foreach my $index ( 0 .. $max_procs-1 ) {
    my $cmd = $$cli_to_execute[$index];

    # Forks and returns the pid for the child:
    my $pid = $pm->start($index) and next;

    my $out = &cli_command( $cmd ) unless $pid; ### This code is the child process

    $pm->finish($index, \$out); # pass an exit code to finish
}

print "Waiting for children...\n";
$pm->wait_all_children;
0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜