URL parsing error [BeautifulSoup]
I'm trying to get a list of href links from website pages; however my code is not working properly. The code is appending when it shouldn't be to the urlList
. It is also duplicating href links.
import urllib2
from BeautifulSoup import BeautifulSoup
response = urllib2.urlopen("http://www.gamefaqs.com")
html = response.read()
soup = BeautifulSoup(html)
doNotProcessList = ["gamespot.com", "cnet.com", "gamefaqs.com"]
urlList = []
for link in soup.findAll('a'):
for bad in doNotProcessList:
if bad not in link['href']:
urlList.append(link['href'])
print urlList
Example error:
[u'http://cbsiprivacy.custhelp.com/app/answers/detail/a_id/1272/', u'http://cbsiprivacy
.custhelp.com/app/answers/detail/a_id/1272/', u'http://www.cbsinteractive.com/terms_of_use.php?tag=ft', u'htt开发者_StackOverflowp://www
.cbsinteractive.com/terms_of_use.php?tag=ft', u'http://www.cbsinteractive.com/terms_of_use.php?tag=ft', u'http://m.g
amefaqs.com/?mob_on=1', u'http://m.gamefaqs.com/?mob_on=1']
The error has to do with "not" in the if statement as removing the not will result in only the bad items being stored in the list like so:
[u'http://membership.gamefaqs.com/1328-4-46.html', u'http://www.gamefaqs.com/user/register.html', u'http://www.games
pot.com/6316274', u'http://www.gamespot.com/6316274', u'http://www.gamespot.com/6316489', u'http://www.gamespot.com/
6316489', u'http://www.gamespot.com/6316225', u'http://www.gamespot.com/6316225', u'http://www.gamespot.com/features
/index.html', u'http://www.gamespot.com/news/6322016.html', u'http://www.gamespot.com/news/6322019.html', u'http://w
ww.gamespot.com/news/6322017.html', u'http://www.gamespot.com/news/6322010.html', u'http://www.gamespot.com/news/632
1996.html', u'http://www.gamespot.com/news/index.html', u'http://www.gamespot.com/features/6314339/index.html', u'ht
tp://www.gamespot.com/features/6313939/index.html', u'http://www.gamespot.com/features/6309202/index.html', u'http:/
/www.gamespot.com/features/6320393/index.html', u'http://www.gamespot.com/features/6162248/index.html', u'http://www
.gamespot.com/gameguides.html', u'http://www.gamespot.com/downloads/index.html', u'http://www.gamespot.com/news/inde
x.html', u'http://www.gamespot.com/pc/index.html', u'http://www.gamespot.com/xbox360/index.html', u'http://www.games
pot.com/wii/index.html', u'http://www.gamespot.com/ps3/index.html', u'http://www.gamespot.com/psp/index.html', u'htt
p://www.gamespot.com/ds/index.html', u'http://www.gamespot.com/ps2/index.html', u'http://www.gamespot.com/gba/index.
html', u'http://www.gamespot.com/mobile/index.html', u'http://www.gamespot.com/cheats.html', u'http://www.gamespot.c
om/forums/index.html', u'http://www.gamespot.com/', u'http://www.gamefaqs.com/features/help/', u'http://sitemap.game
faqs.com/', u'http://www.gamefaqs.com/features/aboutus.html', u'http://reviews.cnet.com/Music/2001-6450_7-0.html', u
'http://reviews.cnet.com/Cell_phones/2001-3504_7-0.html', u'http://reviews.cnet.com/Digital_cameras/2001-6501_7-0.ht
ml', u'http://reviews.cnet.com/Notebooks/2001-3121_7-0.html', u'http://reviews.cnet.com/Handhelds/2001-3127_7-0.html
', u'http://reviews.cnet.com/4521-6531_7-5021436-3.html', u'http://reviews.cnet.com/Web_hosting/2001-6540_7-0.html',
u'http://clearance.cnet.com', u'http://shopper.cnet.com/4520-5-6276184.html', u'http://www.cnet.com', u'http://www.
gamespot.com', u'http://www.gamespot.com/cheats.html', u'http://www.cnet.com/apple-iphone.html', u'http://www.gamesp
ot.com/reviews.html', u'http://reviews.cnet.com/laptops', u'http://download.cnet.com/windows/antivirus-software/', u
'http://m.gamefaqs.com/?mob_on=1']
list comprehension FTW:
[link['href'] for link in soup.findAll('a')
if not any(bad in link['href'] for bad in doNotProcessList)]
And, for readability...
def condition(x):
return not any((bad in x) for bad in doNotProcessList)
[link['href'] for link in soup.findAll('a') if condition(link['href'])]
精彩评论