How to extract webdata with Python, BeautiflSoup and mechanize from table

I would like to extract the data from the table on this site: http://www.pgatour.com/r/stats/info/xm.html?101 And then save it as .csv and bring it into iWorks Numbers sheet. I've been trying with Python, BeautifulSoup and mechanize. I've been trying without knowledge by looking at other examp开发者_运维知识库les, but without success. I have come this far:

from BeautifulSoup import BeautifulSoup, SoupStrainer
from mechanize import Browser
import re
br = Browser()
response = br.open("http://www.pgatour.com/r/stats/info/xm.html?101").read()

Then I look at the code with firebug and I'm guessing I need to parse the data that is between the <tbody> and </tbody>. But I don't know how to do this. Any help much appreciated.

In the main page, the tour stats seems to be being populated by JavaScript <div class="tourViewData"> ... populateDDs(); BS does not parse Javascript, see the many other SO questions on that. (I don't know how to solve that part. At worst, select and save that HTML selection as a local html file, as a workaround.)

First, set up s as a BeautifulSoup object for that URL (I used twill not raw mechanize, put your mechanize equivalent here):

from BeautifulSoup import BeautifulSoup, SoupStrainer
#from mechanize import Browser
from twill.commands import *
import re

go("http://www.pgatour.com/r/stats/info/xm.html?101")
s = BeautifulSoup(get_browser().get_html())

Anyway the table of stats you're looking for is the table tagged with <tbody><tr class="tourStatTournHead">. Just to make things a bit wacky, the tag attribute in its rows is alternately defined as <tr class="tourStatTournCellAlt" or <tr class="".... We should search for the first <tr class="tourStatTournCellAlt", then process every <tr> in the table after that, except the header rows (<tr class="tourStatTournHead">).

To iterate through the rows:

tbl = s.find('table', {'class':'tourStatTournTbl'})

def extract_text(ix,tg):
    if ix==2: # player name field, may be hierarchical
        tg = tg.findChildren()[0] if (len(tg.findChildren())>0) else tg
    return tg.text.encode()

for rec in tbl.findAll('tr'): # {'class':'tourStatTournCellAlt'}):
    # Skip header rows
    if (u'tourStatTournHead' in rec.attrs[0]):
        continue        
    # Extract all fields
    (rank_tw,rank_lw,player,rounds,avg,tot_dist,tot_drives) = \
        [extract_text(i,t) for (i,t) in enumerate(rec.findChildren(recursive=False))]
    # ... do stuff

We add a helper function for player name (it may or may not be hierarchical, if it has a Titleist logo embedded in it.) Probably you want to convert most fields to int() except player(string) and avg(float); if so, remember to strip the optional 'T' (for tied) from the the rank fields, and strip the comma from tot_dist.