Reduce PHP IO load

I have a PHP script that serves alot of smaller files (>100,000) with sizes up to 10mb. It basically loads the requested file into memory and serves it to the client. Because of access control I cannot serve these files by apache directly and need a script wrapped around it.

If there is high traffic (>150mbit) my hdd开发者_C百科 is heavily used and represents a limit for scaling. I had the idea that I could use memcached to reduce the hdd load since I have 10gig of ram available but memcached has a max item size of 1MB. Then I thought I could use PHP-APC but its behaviour if the cache runs out of memory (complete reset) isn't acceptable.

What would you do to reduce the IO load?

Thanks

What would you do to reduce the IO load?

I have never worked with it myself, but the X-Sendfile method may be helpful for taking away a bit of the load. It passes the task of actually serving the file back to Apache.

I think you can't do this unless you have 2 HDD which would split these files.

I would use PHP-APC to load these files into the cache.

apc_add(), apc_fetch() and apc_delete() are what you want. You can ensure you don't overflow by using apc_cache_info() to determine free memory levels. You can also set apc.user_ttl INI setting to prevent total cache clearing on fill.

Set things up on a test server, subject it to high load (with ab or the like) and check your stats using apc.php. Tweak, tweak and tweak some more!

You could use a CDN that supports access control.

If you want to continue serving it yourself, though, there are various approaches you could take. You'll always want to avoid serving the file through PHP though, because that is your bottleneck. None of these are very elegant though.

• Store the files outside of the HTTP root and generate a new symlink every X minutes. The symlinks are deleted after Y time. Your PHP authentication script would then simply redirect the user to the URL in which the (temporarily valid) symlink exists. Very short PHP execution time, files served by Apache.
• Keep the files inside the HTTP root, but change the rewrite rules in a .htacess file instead to achieve the same thing.

Reduce IO load by storing the most frequently accessed files in a ramdisk, integrate them with the regular file system by some mounting magic or symlinks and then let Apache handle the rest.

You need either mod_xsendfile for Apache2 or nginx with X-Accel-Redirect. There is also similar a solution for lighttpd. Nginx can also serve from memcached.

If you're thinking about storing frequently used files in tmpfs, don't. That's not a real solution because even if you serve files right from the disk subsequent requests will hit the system cache and you'll get similar to tmpfs speed.