Storage of thousands of files in Rails

I am working on a Rails app which r开发者_高级运维elies on large collections (1000's of files) of 50MB+ TIFF files. The trouble I'm having is determining how to handle the processing. In an older version of the app, the user had to specify a path (to preferably a mounted SAN drive in the server) in the config file, which became the root for all TIFF collections, and this path was used by the app to do all the processing.

The app needs to:

• Do batch processing (creating JPG, resizing, adding text labels, etc using RMagick) of TIFFs given a path, like the old version. We will still be requiring the user to specify a path to serve as root for all projects.

• Allow users to add new files to TIFF collections in two ways.

  • Add new file(s) to the root path like above, which will be detected by the app and processed

  • Upload files via the web user interface, which then gets stored in the root path.

I wonder if this is an acceptable way to achieve this. The reason we need the root path in the config is because the file sizes simply are too big to upload everything from the Web interface. Imagine how long it would take to upload 1,000 TIFF files, each at least 50MB~100MB in size. I cannot think of any alternatives to handle it. Any ideas will be appreciated.

Instead of directly referencing a local PATH on the server where Rails app is running, it is better to use Capistrano to deploy your application, and symlink a folder under public to your file root.

For example, in rails root:

ln -fs /var/lib/huge_file_collection public/files

Then Rails app will always have a consistent location for files (which can be stored in constants), and Capistrano can be used to dynamically link whichever location you use to a correct path inside rails app.

Of course carrier-wave can then use public/files folder as the root for uploaded files too, if that's desired.