WWW::CURL::Multi module gives Write Error
I'm trying to download a bunch of files on the web in parallel with the Perl's WWW::Curl::Multi module but I'm receiving the following errors:
Use of uninitialized value in ref-to-glob cast at AA.pm line 17.
syswrite() on unopened filehandle at AA.pm line 17.
Use of uninitialized value in
subroutine entry at AA.pm line 60.
I'll appreciate if someone can figure out why those errors come from.
package AA;
use strict;
use warnings;
use WWW::Curl::Easy;
use WWW::Curl::Multi;
use Data::Dumper;
sub new {
my $class = shift;
my $self = {};
bless $self, $class;
return $self;
}
sub tofile {
return syswrite($_[1], $_[0]); #This is where first and the second error occurs
}
sub downloadfiles{
my $self = shift;
my $files = shift;
my %easy;
my $curl = WWW::Curl::Easy->new;
my $active_handles = 0;
my $curlm = WWW::Curl::Multi->new;
my $dir = "dl/";
my $fh;
foreach my $file (@$files) {
my $curl_id = $active_handles + 1; # This should be a handle unique id.
my $code = 0;
$easy{$curl_id} = $curl;
$code+= $curl->setopt(CURLOPT_PRIVATE, $curl_id);
#Open the filehandle
open($fh, ">$dir$curl_id") or die "\nopen: $!\n\n";
binmode $fh;
# do the usual configuration on the handle
$code+= $curl->setopt(CURLOPT_FILE, *$fh);
$code+= $curl->setopt(CURLOPT_FAILONERROR, 1);
$code+= $curl->setopt(CURLOPT_HEADER, 1);
$code+= $curl->setopt(CURLOPT_CONNECTTIMEOUT, 2);
$code+= $curl->setopt(CURLOPT_URL, $file);
$code+= $curl->setopt(CURLOPT_WRITEFUNCTION, \&tofile);
$code+= $curl->setopt(CURLOPT_NOPROGRESS, 1);
$code+= $curl->setopt(CURLOPT_VERBOSE, 0);
$code+= $curl->setopt(CURLOPT_HEADER, 0开发者_JAVA技巧);
if ($code ne 0) {
die("Failed to initialize curl");
}
# Add some easy handles
$curlm->add_handle($curl);
$active_handles++;
}
print "\nActive handles: ".$active_handles."\n";
while ($active_handles) {
my $active_transfers = $curlm->perform; #This is where second error comes from
if ($active_transfers != $active_handles) {
while (my ($id, $return_value) = $curlm->info_read) {
print $id;
if ($id) {
$active_handles--;
my $actual_easy_handle = $easy{$id};
delete $easy{$id};
}
}
}
}
close $fh;
}
1;
I see you initialize a filehandle $fh
to store the downloaded data, but I don't see how that value ever gets passed to your tofile
function. Do you need to say
$code+= $curl->setopt(CURLOPT_FILE, $fh); # or
$code+= $curl->setopt(CURLOPT_FILE, *$fh);
instead of
$code+= $curl->setopt(CURLOPT_FILE, *TBZ2); # whatever TBZ2 is
?
What do you see if you put an extra line in tofile
that prints out the inputs (@_
) to that function?
I've realized that the problem was not about the script, it was rather about the old BSD machine(with the old version of perl) I was using.
You can test the code with the following lines:
my @rrds=qw(http://www.google.com http://www.stackoverflow.com http://www.reddit.com);
print Dumper(@rrds);
my $fDl = AA->new;
$fDl->downloadfiles(\@rrds);
But there is another bug in that script after being able to run it, it outputs the contents to only the last file. Because we are changing the values of $fh. You can fix by defining $fh locally in the for loop and put it in a hash. When exiting the subroutine close the fh's in the hash.
精彩评论