<Web Scraping>Elements not visible in HTML?

I'm trying to write a python web scraper that takes a pandora account and gets all the stations from it.

However, the stations do not immediately all show up, and i need to click the show all button to view all of the stations. Moreover, even after i click the show all, the source code remains unchanged!

My question is where is the html that displays these extra elements that are seemingly invisible?

Example)

if you go to http://www.pandora.com/people/nenadbach#tbl_stations_table,all (the #tbl_stations_table,all makes all the stations show up; this is where the "show all" button takes you)

And view source, the stations after Th开发者_运维百科e Girl From Ipanema Radio arent stored in the immediate source

Thanks for the help!

If you view the source from Firebug (if you use Firefox) or Inspector (if you use Safari or Chrome) you can see that the data is there. It's most likely being pulled in via ajax (JavaScript).

You would either need a scraper that understands JavaScript or to find the http ajax calls its making and call them yourself. The call that you are probably looking for is:

http://www.pandora.com/favorites/profile_tablerows_station.vm?webname=nenadbach&countRowsOnBrowser=10&countRowsNeeded=25

Note that mostly likely this is using a cookie to detect who you are and what list to show.