开发者

Compare two files with English and Japanese data

I am asked to compare two files which contain some Japanese and English informaion. I use a slow approach, read two files and put them in two arrays, then use while loop to check whether the File_B's TERMS are WITHIN the lines of File_A.

File_A:

This is a boy.

ハンカ

She is a girl.

I am a man.

This apple is big.

That orange is small.

File_B:

is a

ハンカ

small

开发者_如何转开发

Output:

I am a man.

This apple is big.

The program I wrote works fine. However, the files I am processing are very huge. The running time is so slow if I use looping like this. My friend told me that I can put the files into hash and it will run far much quicker. I can see the point of using it but the files I have are random and also the terms in File_B can be in any part of File_A. Some previous posts suggest that I can use split the lines of File_A, put the line in hash and compare it with File_B (put File_B in the other hash as well). However, I do not know how to do it besides using ~// to check whether the line got the term.

open(A_FILE, "<", "FILE_A.txt");
my(@a_lines) = <A_FILE>; # read file into list
close(A_FILE);
open(B_FILE, "<", "FILE_B.txt");
my(@b_lines) = <B_FILE>; # read file into list
my($b_lines);
close(B_FILE);
open(my $out, ">",  "Useful.txt") or die "Can't open Useful.txt: $!";
$number = @b_lines;

foreach $a_line (@a_lines) # loop thru list
{
   $found = 0;
   my $sentence = $a_line;
   $i = 0;
   chomp($sentence);
   while (($i <= $number-1) and ($found == 0)){
       chomp($b_lines[$i]);
       if ($sentence =~ /$b_lines[$i]/){
           $found = 1;
       }
       $i++;
   }

   if ($found == 1) {
       print $out $sentence."\n";
   }
}


I don't see how a hash table is going to help you search substrings. It's good for exact matches, though.

If your memory is not a constraint, maybe you could build a suffix tree from all the File A entries, which would give you very fast O(N) search times on File B entries.


If you are running on linux you could write a shell script that sorts the files and then uses the 'uniq' program. Sorting huge files first allows you to compare them without reading the entire files into memory.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜