python scraping by getting urls dynamic way

I am new to the world of data scraping,previously used python for web and desktop app dev开发者_C百科elopment. I am just wondering,if there is any way to get the urls from a page then look into it for specific information like,phone no,address etc.

Currently I am using BeautifulSoup and built method where I am telling the urls as a parameter of the methods.

The site I am scraping large and its really tough to pass the specific url for each page.

Any suggestion to make it faster and self driven?

Thanks in advance.

You can use Scrapy. It simplifies both crawling and parsing (it uses libxml2 for parsing by default).

Use a more efficient HTML parser, like lxml. See here for performance comparisons of various Python parsers.