fast parsing links out of a page in python

I need to parse a large number of pages (say 1000) and replace the links with tiny开发者_运维百科url links.

right now i am doing this using a regex

href_link_re = re.compile(r"<a[^>]+?href\s*=\s*(\"|')(.*?)\1[^>]*>", re.S)

but its not fast enough.

i am thinking so far

1. state machine (the success of this will depend on my ability to write clever code)
2. using an html parser

Can you suggest faster ways?

EDIT: You would think that an html parser would be faster than regex, but in my tests it is not:

from BeautifulSoup import BeautifulSoup, SoupStrainer

import re
import time

__author__ = 'misha'

regex = re.compile(r"<a[^>]+?href\s*=\s*(\"|')(.*?)\1[^>]*>", re.S)

def test(text, fn, desc):

    start = time.time()
    total  = 0
    links = [];
    for i in range(0, 10):
        links = fn(text)
        total += len(links)
    end = time.time()
    print(desc % (end-start, total))
   # print(links)

def parseRegex(text):
    links  = set([])
    for link in regex.findall(text):
        links.add(link[1])
    return links

def parseSoup(text):
    links = set([])
    for link in BeautifulSoup(text, parseOnlyThese=SoupStrainer('a')):
        if link.has_key('href'):
            links.add(link['href'])

    return links



if __name__ == '__main__':
    f = open('/Users/misha/test')
    text = ''.join(f.readlines())
    f.close()

    test(text, parseRegex, "regex time taken: %s found links: %s" )
    test(text, parseSoup, "soup time taken: %s found links: %s" )

output:

regex time taken: 0.00451803207397 found links: 2450
soup time taken: 0.791836977005 found links: 2450

(test is a dump of the wikipedia front page)

i must be using soup badly. what am i doing wrong?

LXML is probably your best bet for this task. See Beautiful Soup vs LXML Performance. Parsing links is easy in LXML and it's fast.

root = lxml.html.fromstring(s)
anchors = root.cssselect("a")
links = [a.get("href") for a in anchors]

Parsing using regexp's very bad idea, because of speed and regexp exponential time problem. Instead you can use parsers for xhtml. The best is LXML. Or you can write parser specially for this purpose with LL,LR parsers. For example: ANTLR,YAPPS,YACC,PYBISON, etc