python removing duplicates

In an array I have the foll开发者_StackOverflowowing tuples:

  ('0000233/02', 50.0, None, None, None, None, 'Yes') 
  ('0000233/02', 200.0, None, None, None, None, 'Yes') 

if im iterating through the list, how could I eliminate duplicates based solely on the first element?

Put them in a dict using the first element as the key. If you check before adding then you'll get the first item with that key, otherwise you'll get the last.

Look first: http://docs.python.org/faq/programming.html#how-do-you-remove-duplicates-from-a-list

>>> l=[('0000233/02', 50.0, None, None, None, None, 'Yes'), ('0000233/02', 200.0, None, None, None, None, 'Yes') ]
>>> dic={}
>>> for i in l: dic[i[0]]=i
...   
>>> dic
{'0000233/02': ('0000233/02', 200.0, None, None, None, None, 'Yes')}
>>> list(dic.values())
[('0000233/02', 200.0, None, None, None, None, 'Yes')]

The ad-hoc solution:

def unique_elem0( iterable ):
    seen = set()
    seen_add = seen.add
    for element in iterable:
        key = element[0]
        if key not in seen:
            seen_add(key)
            yield element

print list(unique_elem0(lst))

The "copy code from the itertools receipes" solution:

def unique_everseen(iterable, key=None):
    "List unique elements, preserving order. Remember all elements ever seen."
    # unique_everseen('AAAABBBCCDAABBB') --> A B C D
    # unique_everseen('ABBCcAD', str.lower) --> A B C D
    seen = set()
    seen_add = seen.add
    if key is None:
        for element in ifilterfalse(seen.__contains__, iterable):
            seen_add(element)
            yield element
    else:
        for element in iterable:
            k = key(element)
            if k not in seen:
                seen_add(k)
                yield element

from operator import itemgetter        
print list(unique_everseen(lst, key=itemgetter(0)))

One slightly different way if your input comes sorted (or at least, with the duplicates all clumped together) is to use itertools.groupby:

import itertools, operator

def filter_duplicates(items):
    for key, group in itertools.groupby(items, operator.itemgetter(0)):
        yield next(group)

This picks the first item of every clump of duplicates (grouping by first item). This is more efficient than the set/dict based approaches, since there's no extra structure needed, and preserves the order of the sequence. It does however depend on the duplicates coming in batches - if they can appear anywhere in the stream, use one of the other methods.

Quick way: create a dictionary, using the element you want to use to compare as key.

# This will leave the last tuple found with that 1st value in the dict:
d = {}
for t in tuples:
    d[t[0]] = t # or .set()

# This will leave the first tuple found, instead of the last:
d = {}
for t in tuples:
    d.setdefault(t[0], t) # setdefault sets the value if it's missing.

If you DO NOT care about the order of the elements in the after the first, this is fast and easy:

>>> t1= ('0000233/02', 50.0, None, None, None, None, 'Yes')
>>> t2= ('0000233/02', 200.0, None, None, None, None, 'Yes')
>>> t1=(t1[0],)+tuple(set(t1[1:]))
>>> t2=(t2[0],)+tuple(set(t2[1:]))
>>> t1
('0000233/02', 50.0, None, 'Yes')
>>> t2
('0000233/02', 200.0, 'Yes', None)

If you DO care about the order:

>>> t2= ('0000233/02', 200.0, None, None, None, None, 'Yes')
>>> nd=[]
>>> garbage=[nd.append(i) for i in t2 if not nd.count(i)]
>>> t2=tuple(nd)
>>> t2
('0000233/02', 200.0, None, 'Yes')