Retrieving and scraping large amounts of data from third party websites

OI want to retrieve and scrape large amounts of data about users from a third party website. Once the person logs in, I get their data from the third party website and scrape it. How do I do this most effectively as to prevent a significant load on the server and to make 开发者_JAVA百科sure pages don't load too slow? I was thinking of setting up a background process that retrieves the data and setting the process to low priority. Do you guys have other tips for me?

I'm using PHP and Mysql.

Data Scraping - Wikipedia

Is this the Data Scraping you are talking about? If yes, we will probably need more details to be able to help.

• How do you know, a user logged in?
• What kind of access do you have to the 3rd party system?

In order to accurately answer, more information is needed. To point you in the correct direction, you should use cURL to load the webpage into a string. Then use PHPs DOM parser to parse the page. If you are looking for specific elements in the page, DOM XPATH queries will be useful (http://php.net/manual/en/class.domxpath.php). Alternatively, you can load the page directly using DOM->LoadHTML() but there are far less options.