using AWK how to remove these kind of Duplicates?

I am new to AWK, I have some basic ideas in AWK. I want to remove duplicates in a file, for example:

    0008.ASIA. NS AS2.DNS.ASIA.CN.
    0008.ASIA. NS AS2.DNS.ASIA.CN.
    ns1.0008.asia. NS AS2.DNS.ASIA.CN.
    www.0008.asia. NS AS2.DNS.ASIA.CN.
    anish.asia NS AS2.DNS.ASIA.CN.
    ns2.anish.asia NS AS2.DNS.ASIA.CN
    ANISH.asia. NS AS2.DNS.ASIA.CN.

This is a sample file, from that using this command I got the output like this:

awk 'BEGIN{IGNORECASE=1}/^[^ ]+asia/ { gsub(/\.$/,"",$1);split($1,a,".")} length(a)==2{b[$1]++;}END{for (x in b)print x}'

0008.ASIA.

anish.asia.

ANISH.asia

But I want output like this

  008.ASIA
  anish.asia

or

008.ASIA
ANISH.asia

How do I remove these kind of duplicates?

Thanks in Advance Anish kumar.V

Thanks for your immediate reponse, Actually I wrote a complete script in bash, now I am in final stage. How to invoke python in that :-(

#!/bin/bash

current_date=`date +%d-%m-%Y_%H.%M.%S`
today=`date +%d%m%Y`
yesterday=`date -d 'yesterday' '+%d%m%Y'`
RootPath=/var/domaincount/asia/
MainPath=$RootPath${today}asia
LOG=/var/tmp/log/asia/asiacount$current_date.log

mkdir -p $MainPath
echo Intelliscan Process started for Asia TLD $current_date 

exec 6>&1 >> $LOG

#################################################################################################
## Using Wget Downloading the Zone files it will try only one time
if ! wget --tries=1 --ftp-user=USERNAME --ftp-password=PASSWORD ftp://ftp.anish.com:21/zonefile/anish.zone.gz
then
    echo Download Not Success Domain count Failed With Error
    exit 1
fi
###The downloaded file in Gunzip format from that we need to unzip and start the domain count process####
gunzip asia.zone.gz > $MainPath/$today.as开发者_高级运维ia

###### It will start the Count #####
awk '/^[^ ]+ASIA/ && !_[$1]++{print $1; tot++}END{print "Total",tot,"Domains"}' $MainPath/$today.asia > $RootPath/zonefile/$today.asia
awk '/Total/ {print $2}' $RootPath/zonefile/$today.asia > $RootPath/$today.count

a=$(< $RootPath/$today.count)
b=$(< $RootPath/$yesterday.count)
c=$(awk 'NR==FNR{a[$0];next} $0 in a{tot++}END{print tot}' $RootPath/zonefile/$today.asia $RootPath/zonefile/$yesterday.asia)

echo "$current_date Count For Asia TlD $a"
echo "$current_date Overall Count For Asia TlD $c"
echo "$current_date New Registration Domain Counts $((c - a))"
echo "$current_date Deleted Domain Counts $((c - b))"

exec >&6 6>&-
cat $LOG | mail -s "Asia Tld Count log" 07anis@gmail.com

In that

 awk '/^[^ ]+ASIA/ && !_[$1]++{print $1; tot++}END{print "Total",tot,"Domains"}' $MainPath/$today.asia > $RootPath/zonefile/$today.asia

in this part only now I am searching how to get the distinct values so any suggestions using AWK is better for me. Thanks again for your immediate response.

kent$  cat a
0008.ASIA. NS AS2.DNS.ASIA.CN.
0008.ASIA. NS AS2.DNS.ASIA.CN.
ns1.0008.asia. NS AS2.DNS.ASIA.CN.
www.0008.asia. NS AS2.DNS.ASIA.CN.
anish.asia NS AS2.DNS.ASIA.CN.
ns2.anish.asia NS AS2.DNS.ASIA.CN
ANISH.asia. NS AS2.DNS.ASIA.CN.


kent$  awk -F' NS' '{ gsub(/\.$/,"",$1);split($1,a,".")} length(a)==2{b[tolower($1)]++;}END{for (x in b)print x}' a
anish.asia
0008.asia

btw, it is interesting, that I gave you a solution at http://www.unix.com/shell-programming-scripting/167512-using-awk-how-its-possible.html, and you add something new in your file, then I added the tolower() function here. :D

By putting your AWK script into a separate file, you can tell what's really going on. Here's a simple approach to your "filter out the duplicates" problem:

# For each line in the file
{

  # Decide on a unique key (eg. case insensitive without trailing period)
  unique_key = tolower($1)
  sub(/\.$/, "", unique_key)

  # If this line isn't a duplicate (it hasn't been found yet)
  if (!(unique_key in already_found)) {

    # Mark this unique key as found
    already_found[unique_key] = "found"

    # Print out the relevant data
    print($1)
  }
}

You can run AWK files by passing the -f option to awk.

If the above script isn't recognizable as an AWK script, here it is in inline form:

awk '{ key = tolower($1); sub(/\.$/, "", key); if (!(key in found)) { found[key] = 1; print($1) } }'

Or, just use the shell:

echo '    0008.ASIA. NS AS2.DNS.ASIA.CN.
    0008.ASIA. NS AS2.DNS.ASIA.CN.
    ns1.0008.asia. NS AS2.DNS.ASIA.CN.
    www.0008.asia. NS AS2.DNS.ASIA.CN.
    anish.asia NS AS2.DNS.ASIA.CN.
    ns2.anish.asia NS AS2.DNS.ASIA.CN
    ANISH.asia. NS AS2.DNS.ASIA.CN.' |
while read domain rest; do
    domain=${domain%.}
    case "$domain" in
        (*.*.*) : ;;
        (*.[aA][sS][iI][aA]) echo "$domain" ;;
    esac
done |
sort -fu

produces

0008.ASIA
anish.asia

Don't use AWK. Use Python

import readlines
result= set()
for line in readlines:
    words = lines.split()
    if "asia" in words[0].lower():
        result.add( words[0].lower() )
for name in result:
    print name

That might be easier to work with than AWK. Yes. It's longer. But it may be easier to understand.

Here is an alternative solution. Let sort create your cased-folded and uniq list (and it will be sorted!)

  {
   cat - <<EOS
   0008.ASIA. NS AS2.DNS.ASIA.CN.
   0008.ASIA. NS AS2.DNS.ASIA.CN.
   ns1.0008.asia. NS AS2.DNS.ASIA.CN.
   www.0008.asia. NS AS2.DNS.ASIA.CN.
   anish.asia NS AS2.DNS.ASIA.CN.
   ns2.anish.asia NS AS2.DNS.ASIA.CN
   ANISH.asia. NS AS2.DNS.ASIA.CN.

EOS
 } |   awk '{
      #dbg print "$0=" $0
      targ=$1
      sub(/\.$/, "", targ)
      n=split(targ,tmpArr,".")
      #dbg print "n="n
      if (n > 2) targ=tmpArr[n-1] "." tmpArr[n]
      print targ 
     }' \
 | sort -f -u

output

0008.ASIA
anish.asia

Edit: fixed sort -i -u to sort -f -u. Many other unix utilties use '-i' to indcate 'ignorecase'. My test showed me I need to fix it, and I forgot to fix the final posting.