Find and append each reference to a html link - Python

I have a HTML file I g开发者_如何转开发ot from Wikipedia and would like to find every link on the page such as /wiki/Absinthe and replace it with the current directory added to the front such as /home/fergus/wikiget/wiki/Absinthe so for:

<a href="/wiki/Absinthe">Absinthe</a>

becomes:

<a href="/home/fergus/wikiget/wiki/Absinthe">Absinthe</a>

and this is throughout the whole document.

Do you have any ideas? I'm happy to use BeautifulSoup or Regex!

If that's really all you have to do, you could do it with sed and its -i option to rewrite the file in-place:

sed -e 's,href="/wiki,href="/home/fergus/wikiget/wiki,' wiki-file.html

However, here's a Python solution using the lovely lxml API, in case you need to do anything more complex or you might have badly formed HTML, etc.:

from lxml import etree
import re

parser = etree.HTMLParser()

with open("wiki-file.html") as fp:
    tree = etree.parse(fp, parser)

for e in tree.xpath("//a[@href]"):
    link = e.attrib['href']
    if re.search('^/wiki',link):
        e.attrib['href'] = '/home/fergus/wikiget'+link

# Or you can just specify the same filename to overwrite it:
with open("wiki-file-rewritten.html","w") as fp:
    fp.write(etree.tostring(tree))

Note that lxml is probably a better option than BeautifulSoup for this kind of task nowadays, for the reasons given by BeautifulSoup's author.

This is solution using re module:

#!/usr/bin/env python
import re

open('output.html', 'w').write(re.sub('href="http://en.wikipedia.org', 'href="/home/fergus/wikiget/wiki/Absinthe', open('file.html').read()))

Here's another one without using re:

#!/usr/bin/env python
open('output.html', 'w').write(open('file.html').read().replace('href="http://en.wikipedia.org', 'href="/home/fergus/wikiget/wiki/Absinthe'))

You can use a function with re.sub:

def match(m):
    return '<a href="/home/fergus/wikiget' + m.group(1) + '">'

r = re.compile(r'<a\shref="([^"]+)">')
r.sub(match, yourtext)

An example:

>>> s = '<a href="/wiki/Absinthe">Absinthe</a>'
>>> r.sub(match, s)
'<a href="/home/fergus/wikiget/wiki/Absinthe">Absinthe</a>'

from lxml import html

el = html.fromstring('<a href="/wiki/word">word</a>')
# or `el = html.parse(file_or_url).getroot()`

def repl(link):
    if link.startswith('/'):
       link = '/home/fergus/wikiget' + link
    return link

print(html.tostring(el))
el.rewrite_links(repl)
print(html.tostring(el))

Output

<a href="/wiki/word">word</a>
<a href="/home/fergus/wikiget/wiki/word">word</a>

You could also use the function lxml.html.rewrite_links() directly:

from lxml import html

def repl(link):
    if link.startswith('/'):
       link = '/home/fergus/wikiget' + link
    return link

print html.rewrite_links(htmlstr, repl)

I would do

import re

ch = '<a href="/wiki/Absinthe">Absinthe</a>'

r = re.compile('(<a\s+href=")(/wiki/[^"]+">[^<]+</a>)')

print ch
print
print r.sub('\\1/home/fergus/wikiget\\2',ch)

EDIT:

this solution have been said not to capture tags with additional attribute. I thought it was a narrow pattern of string that was aimed, such as <a href="/wiki/WORD">WORD</a>

If not, well, no problem, a solution with a simpler RE is easy to write

r = re.compile('(<a\s+href="/)([^>]+">)')

ch = '<a href="/wiki/Aide:Homonymie" title="Aide:Homonymie">'
print ch
print r.sub('\\1home/fergus/wikiget/\\2',ch)

or why not:

r = re.compile('(<a\s+href="/)')

ch = '<a href="/wiki/Aide:Homonymie" title="Aide:Homonymie">'
print ch
print r.sub('\\1home/fergus/wikiget/',ch)