Compare two files with English and Japanese data
I am asked to compare two files which contain some Japanese and English informaion. I use a slow approach, read two files and put them in two arrays, then use while loop to check whether the File_B's TERMS are WITHIN the lines of File_A.
File_A:
This is a boy.
ハンカ
She is a girl.
I am a man.
This apple is big.
That orange is small.
File_B:
is a
ハンカ
small
开发者_如何转开发Output:
I am a man.
This apple is big.
The program I wrote works fine. However, the files I am processing are very huge. The running time is so slow if I use looping like this. My friend told me that I can put the files into hash and it will run far much quicker. I can see the point of using it but the files I have are random and also the terms in File_B can be in any part of File_A. Some previous posts suggest that I can use split the lines of File_A, put the line in hash and compare it with File_B (put File_B in the other hash as well). However, I do not know how to do it besides using ~// to check whether the line got the term.
open(A_FILE, "<", "FILE_A.txt");
my(@a_lines) = <A_FILE>; # read file into list
close(A_FILE);
open(B_FILE, "<", "FILE_B.txt");
my(@b_lines) = <B_FILE>; # read file into list
my($b_lines);
close(B_FILE);
open(my $out, ">", "Useful.txt") or die "Can't open Useful.txt: $!";
$number = @b_lines;
foreach $a_line (@a_lines) # loop thru list
{
$found = 0;
my $sentence = $a_line;
$i = 0;
chomp($sentence);
while (($i <= $number-1) and ($found == 0)){
chomp($b_lines[$i]);
if ($sentence =~ /$b_lines[$i]/){
$found = 1;
}
$i++;
}
if ($found == 1) {
print $out $sentence."\n";
}
}
I don't see how a hash table is going to help you search substrings. It's good for exact matches, though.
If your memory is not a constraint, maybe you could build a suffix tree from all the File A entries, which would give you very fast O(N)
search times on File B entries.
If you are running on linux you could write a shell script that sorts the files and then uses the 'uniq' program. Sorting huge files first allows you to compare them without reading the entire files into memory.
精彩评论