开发者

Trouble parsing HTML using BeautifulSoup

I'm trying to use BeautifulSoup to parse some HTML in Python. Specifically, I'm trying to create two arrays of soup objects: one for the dates of postings on a website, and one for the postings themselves. However, when I use findAll on the div class that matches the postings, only the initial tag is returned, not the text inside the tag. On the other hand, my code works just fine for the dates. What is going on??

# store all texts of posts
texts = soup.findAll("div", {"class":"quote"})

# store all dates of posts
dates = soup.findAll("div", {"class":"datetab"})

The first line above returns only

<div class="quote">

which is not what I want. The second line returns

<div class="datetab">Feb<span>2</span></div>

wh开发者_如何学Goich IS what I want (pre-refining).

I have no idea what I'm doing wrong. Here is the website I'm trying to parse. This is for homework, and I'm really really desperate.


Which version of BeautifulSoup are you using? Version 3.1.0 performs significantly worse with real-world HTML (read: invalid HTML) than 3.0.8. This code works with 3.0.8:

import urllib2
from BeautifulSoup import BeautifulSoup

page = urllib2.urlopen("http://harvardfml.com/")
soup = BeautifulSoup(page)
for incident in soup.findAll('span', { "class" : "quote" }):
    print incident.contents


That site is powered by Tumblr. Tumblr has an API.


There's a python port of Tumblr that you can use to read documents.

from tumblr import Api

api = Api('harvardfml.com')
freq = {}
posts = api.read()
for post in posts:
   #do something here

for your bogus findAll, without the actual source code of your program it is hard to see what is wrong.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜