Why Beautiful Soup cannot display all <td> data in the tables?
I tried to page scrape wikipedia a week ago. But i could not figure out why Beautiful Soup will only show some string from the table column and show "none" for other table column.
NOTE: the table column all contains data.
My program will extract all table columns with the tag "description". I am trying to extract all the description from the table.
The website I am scraping is: http://en.wikipedia.org/wiki/Supernatural_(season_6)
This is my code:
from BeautifulSoup import BeautifulSoup
import urllib
import sys
from urllib import FancyURLopener
class MyOpener(FancyURLopener):
version = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24'
def printList(rowList):
for row in rowList:
print row
print '\n'
return
url = "http://en.wikipedia.org/wiki/Supernatural_(season_6)"
#f = urllib.urlopen(url)
#content = f.read()
#f.close
myopener = MyOpener()
page = myopener.open(url)
content = page.read()
page.close()
soup = BeautifulSoup(''.join(content))
soup.prettify()
movieList = []
rowListTitle = soup.findAll('tr', 'vevent')
print len(rowListTitle)
#printList(rowListTitle)
for row in rowListTitle:
col = row.next # explain this?
if col != 'None':
col = col.findNext("b")
movieTitle = col.string
movieTuple = (movieTitle,'')
movieList.append(movieTuple)
#printList(movieList)
for row in movieList:
print row[0]
rowListDescription = soup.findAll('td' , 'description')
print len(rowListDescription)
index = 1;
while ( index < len(rowListDescription) ):
description = rowListDescription[index]
print description
print description.string
str = description
print '####################################'
movieList[index - 1] = (movieList[index - 1][0],description)
index = index + 1
I did not paste the output as it is really long. But the output is really weird as it did managed to capture the information in 开发者_高级运维the <td>
but when i do a .string
, it gives me an empty content.
Do all the description strings come up empty? From the documentation:
For your convenience, if a tag has only one child node, and that child node is a string, the child node is made available as tag.string, as well as tag.contents[0].
In this case, the description often have child nodes, i.e.: a <a>
link to another Wikipedia article. This counts as a non-string child node, in which case string
for the description node is set to None
.
精彩评论