How to use python csv module for splitting double pipe delimited data

I have got data which looks like:

"1234"||"abcd"||"a1s1"

I am trying to read and write using Python's csv reader and writer. As the csv module's delimiter is l开发者_如何学编程imited to single char, is there any way to retrieve data cleanly? I cannot afford to remove the empty columns as it is a massively huge data set to be processed in time bound manner. Any thoughts will be helpful.

The docs and experimentation prove that only single-character delimiters are allowed.

Since cvs.reader accepts any object that supports iterator protocol, you can use generator syntax to replace ||-s with |-s, and then feed this generator to the reader:

def read_this_funky_csv(source):
  # be sure to pass a source object that supports
  # iteration (e.g. a file object, or a list of csv text lines)
  return csv.reader((line.replace('||', '|') for line in source), delimiter='|')

This code is pretty effective since it operates on one CSV line at a time, provided your CSV source yields lines that do not exceed your available RAM :)

>>> import csv
>>> reader = csv.reader(['"1234"||"abcd"||"a1s1"'], delimiter='|')
>>> for row in reader:
...     assert not ''.join(row[1::2])
...     row = row[0::2]
...     print row
...
['1234', 'abcd', 'a1s1']
>>>

Unfortunately, delimiter is represented by a character in C. This means that it is impossible to have it be anything other than a single character in Python. The good news is that it is possible to ignore the values which are null:

reader = csv.reader(['"1234"||"abcd"||"a1s1"'], delimiter='|')
#iterate through the reader.
for x in reader:
    #you have to use a numeric range here to ensure that you eliminate the 
    #right things.
    for i in range(len(x)):
        #Odd indexes will be discarded.
        if i%2 == 0: x[i] #x[i] where i%2 == 0 represents the values you want.

There are other ways to accomplish this (a function could be written, for one), but this gives you the logic which is needed.

If your data literally looks like the example (the fields never contain '||' and are always quoted), and you can tolerate the quote marks, or are willing to slice them off later, just use .split

>>> '"1234"||"abcd"||"a1s1"'.split('||')
['"1234"', '"abcd"', '"a1s1"']
>>> list(s[1:-1] for s in '"1234"||"abcd"||"a1s1"'.split('||'))
['1234', 'abcd', 'a1s1']

csv is only needed if the delimiter is found within the fields, or to delete optional quotes around fields