开发者

URL parsing error [BeautifulSoup]

I'm trying to get a list of href links from website pages; however my code is not working properly. The code is appending when it shouldn't be to the urlList. It is also duplicating href links.

import urllib2
from BeautifulSoup import BeautifulSoup

response = urllib2.urlopen("http://www.gamefaqs.com")
html = response.read()
soup = BeautifulSoup(html)

doNotProcessList = ["gamespot.com", "cnet.com", "gamefaqs.com"]

urlList = []

for link in soup.findAll('a'):
    for bad in doNotProcessList:
        if bad not in link['href']:
            urlList.append(link['href'])

print urlList

Example error:

[u'http://cbsiprivacy.custhelp.com/app/answers/detail/a_id/1272/', u'http://cbsiprivacy
.custhelp.com/app/answers/detail/a_id/1272/', u'http://www.cbsinteractive.com/terms_of_use.php?tag=ft', u'htt开发者_StackOverflowp://www
.cbsinteractive.com/terms_of_use.php?tag=ft', u'http://www.cbsinteractive.com/terms_of_use.php?tag=ft', u'http://m.g
amefaqs.com/?mob_on=1', u'http://m.gamefaqs.com/?mob_on=1']

The error has to do with "not" in the if statement as removing the not will result in only the bad items being stored in the list like so:

[u'http://membership.gamefaqs.com/1328-4-46.html', u'http://www.gamefaqs.com/user/register.html', u'http://www.games
pot.com/6316274', u'http://www.gamespot.com/6316274', u'http://www.gamespot.com/6316489', u'http://www.gamespot.com/
6316489', u'http://www.gamespot.com/6316225', u'http://www.gamespot.com/6316225', u'http://www.gamespot.com/features
/index.html', u'http://www.gamespot.com/news/6322016.html', u'http://www.gamespot.com/news/6322019.html', u'http://w
ww.gamespot.com/news/6322017.html', u'http://www.gamespot.com/news/6322010.html', u'http://www.gamespot.com/news/632
1996.html', u'http://www.gamespot.com/news/index.html', u'http://www.gamespot.com/features/6314339/index.html', u'ht
tp://www.gamespot.com/features/6313939/index.html', u'http://www.gamespot.com/features/6309202/index.html', u'http:/
/www.gamespot.com/features/6320393/index.html', u'http://www.gamespot.com/features/6162248/index.html', u'http://www
.gamespot.com/gameguides.html', u'http://www.gamespot.com/downloads/index.html', u'http://www.gamespot.com/news/inde
x.html', u'http://www.gamespot.com/pc/index.html', u'http://www.gamespot.com/xbox360/index.html', u'http://www.games
pot.com/wii/index.html', u'http://www.gamespot.com/ps3/index.html', u'http://www.gamespot.com/psp/index.html', u'htt
p://www.gamespot.com/ds/index.html', u'http://www.gamespot.com/ps2/index.html', u'http://www.gamespot.com/gba/index.
html', u'http://www.gamespot.com/mobile/index.html', u'http://www.gamespot.com/cheats.html', u'http://www.gamespot.c
om/forums/index.html', u'http://www.gamespot.com/', u'http://www.gamefaqs.com/features/help/', u'http://sitemap.game
faqs.com/', u'http://www.gamefaqs.com/features/aboutus.html', u'http://reviews.cnet.com/Music/2001-6450_7-0.html', u
'http://reviews.cnet.com/Cell_phones/2001-3504_7-0.html', u'http://reviews.cnet.com/Digital_cameras/2001-6501_7-0.ht
ml', u'http://reviews.cnet.com/Notebooks/2001-3121_7-0.html', u'http://reviews.cnet.com/Handhelds/2001-3127_7-0.html
', u'http://reviews.cnet.com/4521-6531_7-5021436-3.html', u'http://reviews.cnet.com/Web_hosting/2001-6540_7-0.html',
 u'http://clearance.cnet.com', u'http://shopper.cnet.com/4520-5-6276184.html', u'http://www.cnet.com', u'http://www.
gamespot.com', u'http://www.gamespot.com/cheats.html', u'http://www.cnet.com/apple-iphone.html', u'http://www.gamesp
ot.com/reviews.html', u'http://reviews.cnet.com/laptops', u'http://download.cnet.com/windows/antivirus-software/', u
'http://m.gamefaqs.com/?mob_on=1']


list comprehension FTW:

[link['href'] for link in soup.findAll('a') 
 if not any(bad in link['href'] for bad in doNotProcessList)]

And, for readability...

def condition(x):
    return not any((bad in x) for bad in doNotProcessList)

[link['href'] for link in soup.findAll('a') if condition(link['href'])]
0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜