Stopword removal and save the new file

I have text files which I need to remove stop words from them. I have the stop words stored in a text file. I load the "stop-word" text file into my Perl script and store the stop words in an array called "stops".

Currently I am loading a different set of text files and I am storing them in a separate array then doing a pattern match to see if any of the words are indeed stop words. I can print the stop words and know which ones are occurring in the files but how do I remove them from the text fi开发者_高级运维le and store a new text file so it has no stop words?

i.e Stopwords: the a to of and into

Text File: "The girl was driving and crashed into a man"

Resulting file: girl was driving crashed man

I load the file in:

$dirtoget = "/Users/j/temp/";
opendir( IMD, $dirtoget ) || die("Cannot open directory");`
@thefiles = readdir(IMD);`

foreach $f (@thefiles) {
if ( $f =~ m/\.txt$/ ) {

    open( FILE, "/Users/j/temp/$f" ) or die "Cannot open FILE";

    while (<FILE>) {
        @file = <FILE>;

Here is the pattern matching loop:

  foreach $word(split) {
                foreach $x (@stop) {
                   if  ($x =~ m/\b\Q$word\E\b/) {
                 $word='';
                        print $word,"\n";

Setting $word to be null.

Or I could do:

    $word = '' if exists $stops{$word};

I'm just not sure how I set output file to no longer contain the matching words. Is it stupid to store the words which don't match in an array and output them to a file?

Overwriting the files in-place is possible, but a hassle. The Unix way of doing this is to just output the non-stopwords to standard output (which print does by default), redirect that

./remove_stopwords.pl textfile.txt > withoutstopwords.txt

then proceed with the file withoutstopwords.txt. This also allows the use of the program in a pipeline.

Shorter:

use strict;
use warnings;
use English qw<$LIST_SEPARATOR $NR>;

my $stop_regex 
    = do { 
        local $LIST_SEPARATOR = '\\E|\\Q';
        eval "qr/\\b(\\Q@{stop}\\E)\\b/";
    };
@ARGV = glob( '/Users/j/temp/*.txt' );
while ( <> ) { 
    next unless m/$stop_regex/;
    print "Stop word '$1' found at $ARGV line $NR\n";
}

What do you want to do with these words? If you wanted to replace them then you could do this:

use English qw<$INPLACE_EDIT $LIST_SEPARATOR $NR>;
local $INPLACE_EDIT = 'bak';

...
while ( <> ) { 
    if ( m/$stop_regex/ )
        s/$stop_regex/$something_else/g;
    }
    print;
}

With $INPLACE_EDIT active, perl will dump the print into a '.bak' file and when it moves onto the next file, it will write the .bak to the original file. If that's what you want to do.

You can use the substitution operator to delete words from your files:

use warnings;
use strict;

my @stop = qw(foo bar);
while (<DATA>) {
    my $line = $_;
    $line =~ s/\b$_\b//g for @stop;
    print $line;
}

__DATA__
here i am
with a foo
and a bar too
lots of foo foo food

prints:

here i am
with a
and a  too
lots of   food