开发者

php array , foreach add multiple query , parallel requests and improve performance

I have the following code which I run from CLI so no Apache process..:

//$multiContent is an array of JSON responses 
foreach($multi_Content as $htmlContent)
{

    //get the items from JSON response
    $htmlContent =  $htmlContent['findItems']['0']['searchResult']['0']['item'];

    //get sellerinfo from each item block
   foreach($htmlContent as $item)
     {
        //print_r($item);
        $username = $item['seller']['0']['UserName']['0'];

        //verify if the user already exists in our db
        $number_object = $db->query("SELECT count(*) as number from users WHE开发者_如何学编程RE    username='$username'");
        $number_object = $number_object->fetch_assoc();
        $number = $number_object['number'];
        if($number < 1)
        {
            $db->query("INSERT INTO users(id, username)VALUES('', '$username')");
        }
    }
}  

Is it possible to process all the json responses at once ? I mean without to get each response one by one(with foreach) to get all of them in a single process task and then insert all of them in the database using a parallel technique . I'm processing large amounts of data and I found this part of code performing quite slowly so any advice would be welcome !!! I'm using mysqli


Not sure, but i think that if you use something like this, that application will be faster.

$values = array();
foreach ($number_object -> fetch_assoc() as $data)
{
    $values[] = "(".$objct['number'].",'".$username."')";
    if (count($values) > 9)
    {
        $values_str = implode(',', $values);
        query("INSERT INTO users(id, username) VALUES ".$values_str);
        $values = array();
    }
}


make sure you have done indexing on "username" field.

Also try to do this changes, if this can improve...

//$multiContent is an array of JSON responses 
$usernames = array();
foreach($multi_Content as $htmlContent)
{

    //get the items from JSON response
    $htmlContent =  $htmlContent['findItems']['0']['searchResult']['0']['item'];

    //get sellerinfo from each item block
   foreach($htmlContent as $item)
     {
        //print_r($item);
        $username = $item['seller']['0']['UserName']['0'];

        //verify if the user already exists in our db
        $number_object = $db->query("SELECT count(id) as number from users WHERE    username='$username'");
        $number_object = $number_object->fetch_assoc();
        $number = $number_object['number'];
        if($number < 1)
        {
            $usernames[] = $username;
        }
    }
}  

if(count($usernames) == 0)
exit;

$sql = 'insert into users (username) values ';
foreach ($usernames as $username)
{
      $sql .= "('".mysql_real_escape_string($username)."'),";
}

//remove out extra comma from end of the string
$sql = substr($sql, 0 , -1);

$db->query($sql);

as you have mentioned u are processing large numbers of data, You can insert this $usernames array in the batch of 250 or so. because large query length can forbid insertion.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜