how to handle large page impressions in memcache before flushing to mysql
I'm writing an analytics tool which will essentially log every time a page is hit. The problem I have is the amount of impressions could become very high and I will potentially need to handle millions a day or at least ~300 a second (I've head people say they can achieve 10k per second). My current setup is php logging each time a page is accessed as an insert directly into MySQL innodb table, this is becoming slow and the server appears to miss a hit occasionally which isn't too bad for analytics but not ideal. My idea is to put memcache in the front and log all impr开发者_StackOverflow社区essions to that first and then run a scheduled batching process which takes the impressions from memcache and then puts them into the db at a quiet period e.g. 3am. So my question is
1) Is this the right way to do it? Ideally I do not want to implement any new technology or implement a NoSQL db such as mongo or couch.
2) How would I do this in memcache? As there is no way of finding out all of the keys currently in cache I was thinking of setting a counter which increments on each impression and then a key with the counter as an id containing the data, the batch process would loop from its last position to the counter and process those records and then set its last position to the current counter..
Any help is appreciated
Cheers
I wouldn't create an serperate batch, but instead integratie it in your code.
Note that there is no checking/validation/locking here. Some pseudocode:
$mViews = $oMemcache->get('views');
if ($mViews > 1000) {
$oDatabase->update($mViews);
$oMemcached->replace('views', 1);
}
else {
$oMemcached->increment('views');
}
精彩评论