How to modify lxml autolink to be more liberal?
I am using the autolink function of the great lxml library as documented here: http://lxml.de/api/lxml.html.clean-module.html
My problem is that it only detects urls that start with http://. I would like to use a broader url detection regex like this one: http://daringfireball.net/2010/07/improved_regex_for_matching_urls
I tried to make that regex work with the lxml autolink function without success. I always end up with a:
lxml\html\c开发者_开发问答lean.py", line 571, in _link_text
host = match.group('host')
IndexError: no such group
Any python/regex gurus out there who know how to make this work?
There are two things to do in order to adapt the regexp to lxml's autolink. First wrap the entire url pattern match in a group (?P<body> .. ) - this lets lxml know what goes inside the href="" attribute.
Next, wrap the host part in a (?<host> .. ) group and pass avoid_hosts=[] parameter when you call the autolink function. The reason for this is the regexp pattern you're using doesn't always find a host (sometimes the host part will be None) since it matches partial urls and ambiguous url-like patterns.
I've modified the regexp to include the above changes and given a snippet test case:
import re
import lxml.html
import lxml.html.clean
url_regexp = re.compile(r"""(?i)\b(?P<body>(?:[a-z][\w-]+:(?:/{1,3}|[a-z0-9%])|www\d{0,3}[.]|(?P<host>[a-z0-9.\-]+[.][a-z]{2,4}/))(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’]))""")
DOC = """<html><body>
http://foo.com/blah_blah
http://foo.com/blah_blah/.
http://www.extinguishedscholar.com/wpglob/?p=364.
http://✪df.ws/1234
rdar://1234
rdar:/1234
message://%3c330e7f840905021726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e
What about <mailto:gruber@daringfireball.net?subject=TEST> (including brokets).
bit.ly/foo
</body></html>"""
tree = lxml.html.fromstring(DOC)
body = tree.find('body')
lxml.html.clean.autolink(body, [url_regexp], avoid_hosts=[])
print lxml.html.tostring(tree)
Output:
<html><body>
<a href="http://foo.com/blah_blah">http://foo.com/blah_blah</a>
<a href="http://foo.com/blah_blah/">http://foo.com/blah_blah/</a>.
<a href="http://www.extinguishedscholar.com/wpglob/?p=364">http://www.extinguishedscholar.com/wpglob/?p=364</a>.
<a href="http://%C3%A2%C2%9C%C2%AAdf.ws/1234">http://✪df.ws/1234</a>
<a href="rdar://1234">rdar://1234</a>
<a href="rdar:/1234">rdar:/1234</a>
<a href="message://%3c330e7f840905021726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e">message://%3c330e7f840905021726r6a4ba78dkf1fd71420c1bf6ff@mail.gmail.com%3e</a>
What about <<a href="mailto:gruber@daringfireball.net?subject=TEST">mailto:gruber@daringfireball.net?subject=TEST</a>>
(including brackets).
<a href="bit.ly/foo">bit.ly/foo</a>
</body></html>
You don't really give enough information to be sure, but I bet that you're having escaping issues with the backslashes in Gruber's regex. Try using a raw string, which allows backslashes without escaping, and triple-quotes, which allow you to use quotes in the string without having to escape those either. E.g.
re.compile(r"""(?i)\b((?:[a-z][\w-]+:(?:/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’]))""")
加载中,请稍侯......
精彩评论