Is there any python lib to scrape search engine(S) results?

I am looking for a python library to scrape results from search engines (google, yahoo, bing, etc).

I only found for google -> http://github.com/kevinw/xgoogle/tree/253db7ddc8603a9dcb038ae42684cf3499a22a4b

Does someone knows o开发者_运维问答ne for multiple search engines?

Scrapy is a pretty cool framework for scraping, but you will have code/configure it to work for the sites you want.

It's not too hard to write them. I usually just use php. Look into curl to retrive the page and then the dom object and dom xpath. You can use xpath to select the parts of the result you want.

Xpath is pretty simple if you install firebug and firexpath. I am working on a position checker right now. Same idea but it returns the position of a domain based on a keyword.

All of the answers here are deprecated, use the standard Google API if you want, you can make 1000 requests in 24 hour period for free.

What else can you try:

• Use requests
• Use selenium
• Use the 3rd party google libraries (all deprecated to my knowledge)

But you will eventually get blocked, so better use the Google supported API or any other paid API.

scraper, you can scrape bing,google, baida,yahoo.check link