Separate number/letter tokens in Python

I'm using re.split() to separate a string into tokens. Currently the pattern I'm using as the argument is [^\dA-Za-z], which retrieves alphanumeric tokens from the str开发者_如何学编程ing.

However, what I need is to also split tokens that have both numbers and letters into tokens with only one or the other, eg.

re.split(pattern, "my t0kens")

would return ["my", "t", "0", "kens"].

I'm guessing I might need to use lookahead/lookbehind, but I'm not sure if that's actually necessary or if there's a better way to do it.

Try the findall method instead.

>>> print re.findall ('[^\d ]+', "my t0kens");
['my', 't', 'kens']
>>> print re.findall ('[\d]+', "my t0kens");
['0']
>>>

Edit: Better way from Bart's comment below.

>>> print re.findall('[a-zA-Z]+|\\d+', "my t0kens")
['my', 't', '0', 'kens']
>>>

>>> [x for x in re.split(r'\s+|(\d+)',"my t0kens") if x]
['my', 't', '0', 'kens']

By using capturing parenthesis within the pattern, the tokens will also be return. Since you only want to maintain digits and not the spaces, I've left the \s outside the parenthesis so None is returned which can then be filtered out using a simple loop.

Should be one line of code

re.findall('[a-z]+|[\d]+', 'my t0kens')

Not perfect, but removing space from the list below is easy :-)

re.split('([\d ])', 'my t0kens')
['my', ' ', 't', '0', 'kens']

docs: "Split string by the occurrences of pattern. If capturing parentheses are used in pattern, then the text of all groups in the pattern are also returned as part of the resulting list."