What is automated way to remove text line if duplicated

For example (below), every line is unique except green car

blue car
red car
green car
black 开发者_如何转开发car
white car
yellow car
green car
brown car

All I want is not remove one line of green car... but I want remove both of green car..

in other words, if any text line is duplicated 2 times or more, remove all of them

If your strings are in a file named data in the current directory, the following command line does what you want on *NIX systems:

$ sort < data | uniq -c | awk '{ if ($1 == 1) print }' | cut -d' ' -f8-

EDIT: If you want to save the results back to the input file, you can do:

$ sort < data | uniq -c | awk '{ if ($1 == 1) print }' \
> | cut -d' ' -f8- > data.new && mv data.new data

To process all files in the current directory, you can do:

$ for f in *; do sort < $f | uniq -c | awk '{ if ($1 == 1) print }' \
> | cut -d' ' -f8- > $f.new && mv $f.new $f; done

blue car
red car
black car
white car
yellow car
brown car

Edit: You could do this in php:

$a = array_count_values($mydata);
foreach ($mydata as $key=>$datum) {
  if ($a[$datum] > 1) unset($mydata[$key]);
}

a small groovy script

def list =["blue car","red car","green car","green car","green car","brown car"]
def value =0
def finalList =[]
list.each{
    value = list.count(it)
    finalList.add(it)
    if(value >1){
        finalList.remove(it)
    }
}
def file = new File("dulicateRemoved.txt")
file<<finalList