php array , foreach add multiple query , parallel requests and improve performance
I have the following code which I run from CLI so no Apache process..:
//$multiContent is an array of JSON responses
foreach($multi_Content as $htmlContent)
{
//get the items from JSON response
$htmlContent = $htmlContent['findItems']['0']['searchResult']['0']['item'];
//get sellerinfo from each item block
foreach($htmlContent as $item)
{
//print_r($item);
$username = $item['seller']['0']['UserName']['0'];
//verify if the user already exists in our db
$number_object = $db->query("SELECT count(*) as number from users WHE开发者_如何学编程RE username='$username'");
$number_object = $number_object->fetch_assoc();
$number = $number_object['number'];
if($number < 1)
{
$db->query("INSERT INTO users(id, username)VALUES('', '$username')");
}
}
}
Is it possible to process all the json responses at once ? I mean without to get each response one by one(with foreach) to get all of them in a single process task and then insert all of them in the database using a parallel technique . I'm processing large amounts of data and I found this part of code performing quite slowly so any advice would be welcome !!! I'm using mysqli
Not sure, but i think that if you use something like this, that application will be faster.
$values = array();
foreach ($number_object -> fetch_assoc() as $data)
{
$values[] = "(".$objct['number'].",'".$username."')";
if (count($values) > 9)
{
$values_str = implode(',', $values);
query("INSERT INTO users(id, username) VALUES ".$values_str);
$values = array();
}
}
make sure you have done indexing on "username"
field.
Also try to do this changes, if this can improve...
//$multiContent is an array of JSON responses
$usernames = array();
foreach($multi_Content as $htmlContent)
{
//get the items from JSON response
$htmlContent = $htmlContent['findItems']['0']['searchResult']['0']['item'];
//get sellerinfo from each item block
foreach($htmlContent as $item)
{
//print_r($item);
$username = $item['seller']['0']['UserName']['0'];
//verify if the user already exists in our db
$number_object = $db->query("SELECT count(id) as number from users WHERE username='$username'");
$number_object = $number_object->fetch_assoc();
$number = $number_object['number'];
if($number < 1)
{
$usernames[] = $username;
}
}
}
if(count($usernames) == 0)
exit;
$sql = 'insert into users (username) values ';
foreach ($usernames as $username)
{
$sql .= "('".mysql_real_escape_string($username)."'),";
}
//remove out extra comma from end of the string
$sql = substr($sql, 0 , -1);
$db->query($sql);
as you have mentioned u are processing large numbers of data, You can insert
this $usernames
array in the batch of 250 or so. because large query length can forbid insertion.
精彩评论