Regex to match 'lol' to 'lolllll' and 'omg' to 'omggg', etc

Hey there, I love regular expressions, but I'm just not good at them at all.

I have a list of some 400 shortened words such as lol, omg, lmao...etc. Whenever someone types one of these shortened words, it is replaced with its English counterpart ([laughter], or something to that effect). Anyway, people are annoying and type these short-hand words with the last letter(s) repeated x number of times.

examples: omg -> omgggg, lol -> lollll, haha -> hahahaha, lol 开发者_开发技巧-> lololol

I was wondering if anyone could hand me the regex (in Python, preferably) to deal with this?

Thanks all.

(It's a Twitter-related project for topic identification if anyone's curious. If someone tweets "Let's go shoot some hoops", how do you know the tweet is about basketball, etc)

FIRST APPROACH -

Well, using regular expression(s) you could do like so -

import re
re.sub('g+', 'g', 'omgggg')
re.sub('l+', 'l', 'lollll')

etc.

Let me point out that using regular expressions is a very fragile & basic approach to dealing with this problem. You could so easily get strings from users which will break the above regular expressions. What I am trying to say is that this approach requires lot of maintenance in terms of observing the patterns of mistakes the users make & then creating case specific regular expressions for them.

SECOND APPROACH -

Instead have you considered using difflib module? It's a module with helpers for computing deltas between objects. Of particular importance here for you is SequenceMatcher. To paraphrase from official documentation-

SequenceMatcher is a flexible class for comparing pairs of sequences of any type, so long as the sequence elements are hashable. SequenceMatcher tries to compute a "human-friendly diff" between two sequences. The fundamental notion is the longest contiguous & junk-free matching subsequence.

import difflib as dl
x   = dl.SequenceMatcher(lambda x : x == ' ', "omg", "omgggg")
y   = dl.SequenceMatcher(lambda x : x == ' ', "omgggg","omg")
avg = (x.ratio()+y.ratio())/2.0
if avg>= 0.6: 
    print 'Match!'
else:
    print 'Sorry!'

According to documentation, any ratio() over 0.6 is a close match. You might need to explore tweak the ratio for your data needs. If you need more stricter matching I found any value over 0.8 serves well.

How about

\b(?=lol)\S*(\S+)(?<=\blol)\1*\b

(replace lol with omg, haha etc.)

This will match lol, lololol, lollll, lollollol etc. but fail lolo, lollllo, lolly and so on.

The rules:

1. Match the word lol completely.
2. Then allow any repetition of one or more characters at the end of the word (i. e. l, ol or lol)

So \b(?=zomg)\S*(\S+)(?<=\bzomg)\1*\b will match zomg, zomggg, zomgmgmg, zomgomgomg etc.

In Python, with comments:

result = re.sub(
    r"""(?ix)\b    # assert position at a word boundary
    (?=lol)        # assert that "lol" can be matched here
    \S*            # match any number of characters except whitespace
    (\S+)          # match at least one character (to be repeated later)
    (?<=\blol)     # until we have reached exactly the position after the 1st "lol"
    \1*            # then repeat the preceding character(s) any number of times
    \b             # and ensure that we end up at another word boundary""", 
    "lol", subject)

This will also match the "unadorned" version (i. e. lol without any repetition). If you don't want this, use \1+ instead of \1*.