MySQL query caching: limited to a maximum cache size of 128 MB?

My application is very database intensive so I've tried really hard to make sure the application and the MySQL database are working as efficiently as possible together.

Currently I'm tuning the MySQL query cache to get it in line with the characteristics of queries being run on the server.

query_cache_size is the maximum amount of data that may be stored in the cache and query_cache_limit is the maximum size of a single resultset in the cache.

My current MySQL quer开发者_如何转开发y cache is configured as follows:

query_cache_size=128M
query_cache_limit=1M

tuning-primer.sh gives me the following tuning hints about the running system:

QUERY CACHE
Query cache is enabled
Current query_cache_size = 128 M
Current query_cache_used = 127 M
Current query_cache_limit = 1 M
Current Query cache Memory fill ratio = 99.95 %
Current query_cache_min_res_unit = 4 K
However, 21278 queries have been removed from the query cache due to lack of memory
Perhaps you should raise query_cache_size
MySQL won't cache query results that are larger than query_cache_limit in size

And mysqltuner.pl gives the following tuning hints:

[OK] Query cache efficiency: 31.3% (39K cached / 125K selects)
[!!] Query cache prunes per day: 2300654

Variables to adjust:
    query_cache_size (> 128M)

Both tuning scripts suggest that I should raise the query_cache_size. However, increasing the query_cache size over 128M may reduce performance according to mysqltuner.pl (see http://mysqltuner.pl/).

How would you tackle this problem? Would you increase the query_cache_size despite mysqltuner.pl's warning or try to adjust the querying logic in some way? Most of the data access is handled by Hibernate, but quite a lot of hand-coded SQL is used in the application as well.

The warning issued by mysqltuner.py is actually relevant even if your cache has no risk of being swapped. It is well-explained in the following: http://blogs.oracle.com/dlutz/entry/mysql_query_cache_sizing

Basically MySQL spends more time grooming the cache the bigger the cache is and since the cache is very volatile under even moderate write loads (queries gets cleared often), putting it too large will have an adverse effect on your application performance. Tweak the query_cache_size and query_cache_limit for your application, try finding a breaking point where you have most hits per insert, a low number of lowmem_prunes and keep a close eye on your database servers load while doing so too.

Usually "too big cache size" warnings are issued under assumption that you have few physical memory and the cache itself well need to be swapped or will take resources that are required by the OS (like file cache).

If you have enough memory, it's safe to increase query_cache size (I've seen installations with 1GB query cache).

But are you sure you are using the query cache right? Do have lots of verbatim repeating queries? Could you please post the example of a typical query?

You should be easy on increasing your cache, it is not only a "not that much available mem" thing!

Reading for instance the manual you get this quote:

Be cautious about sizing the query cache excessively large, which increases the overhead required to maintain the cache, possibly beyond the benefit of enabling it. Sizes in tens of megabytes are usually beneficial. Sizes in the hundreds of megabytes might not be.

There are various other sources you can check out!

A non-zero prune rate may be an indication that you should increase the size of your query cache. However, keep in mind that the overhead of maintaining the cache is likely to increase with its size, so do this in small increments and monitor the result. If you need to dramatically increase the size of the cache to eliminate prunes, there is a good chance that your workload is not a good match for the query cache.

So don't just put as much as you can in that query cache!

The best thing, would be to gradually increase the query cache and measure performance on your site. It's some sort of default in performance questions, but in cases like this 'testing' is one of the best things you can do.

Be careful with setting the query_cache_size and limit to high. MySQL only uses a single thread to read from the query cache.

With the query_cache_size set to 4G and query_cache_limit 12M we had a query cache rate of 85% but noticed a recurring spikes in connections.

After changing the query_cache_size to 256M with 64K query_cache_limit the query cache ratio dropped to 50% but the overall performance increased.

Overhead for Query cache is around 10% so I would disable query caching. Usually if you can't get your hit rate over 40 or 50 % maybe query cache isn't right for your database.

I've blog about this topic... Mysql query_cache_size performance here.

Query Cache gets invalidated/flush every time there is an insert, Use InnoDB/cache and avoid query cache or set it to a very small value.