Managing 3000 files in a client PC folder for a C# WPF application

I want to create a folder with around 3000 files on a user PC for a WPF C# application. I am worried whether after I implement, if the lookup of a filename will be too slow. Can someone please suggest whether it should work, or if there is a better alternative? Basically, my main goal is to use Clickonce to deploy the application on any user PC, and start it to run without imposing requirements of downloading database application.

Background: I have created a C# application to find correlated pairs from yahoo finance. It currently downloads quotes for 3000 symbols and stores it in a Mysql database on the user PC. I am interested in creating a new version of the application which does not depend on Mysql database. I am thinking of saving all the quotes in a separate files in a folder on the user PC.Not sure if this is a good idea, or if some 开发者_运维百科other alternatives like creating small object database on the user PC, using some helper class library which manages cache of large arrays, etc. would be ideal.

PS : It is an open source education project, so I cannot afford to buy commercial product.

I'd recommend using (as you've said that you're willing to look at alternatives to the file system) Sql Server Compact 4.0. It's:

• Lightweight
• Doesn't require a separate installer to be run
• Doesn't need a separate service to be running
• Light on memory footprint
• Very easy to code against as you can use DataTable, DataSet and a set of SqlServerCe based SqlConnection / SqlDataAdapter classes that will be very familiar.
• Free!
• A good candidate for upgrading to Sql Server Express in the future if you need to.
• Has tooling available (in Webmatrix, and I think Visual Studio 2010 SP1) that allows you to manipulate the database structure without having to resort to code.

You'll probably be able to achieve a swap-over in a very short space of time as "most" of the work can be achieved by adding a reference to SqlServerCe to your project and performing a search & replace on "MySql" for "SqlServerCe". That's a slight oversimplification, but not by much! =)

How about using SQLite? You wouldn't need to install anything else than your software. Here is a wrapper for .NET.

To answer your question regarding looking up the filename on the drive, that depends on what kind of drive it is. If it is an SSD it will probably not take long at all, if it is a mechanical drive and you are using indexing it will also be fast. However if you don't use indexing and have a slower mechanical drive, it will be slow.

You could also consider a RAM Disk / RAM Drive if you really want performance and want to have it stored on a filesystem. However, you would need to store it on a persistant storage as well.

You could use a simple file-based database such as SQLite.

http://www.sqlite.org/

There is a .NET driver for it as well.

http://code.google.com/p/csharp-sqlite/

I wouldn't advise falling back to the file system in lieu of a database in this case. Have you considered SQL Server Compact Edition or SQLite? Both run in-process, meaning that you don't have to install a database server - it's part of your application. And, of course, they still support SQL.

3000 files will be fine, 300,000 files and you'd have a problem :)

The only way to know for sure is to measure!

Create folders with 10 zero length files , 100 files 1000 files . . . 10,000,000.

Then measure the time it takes to find a file in each of the folders, you'll get a better idea which are "too slow".

P.s. "Too slow" is completely subjective. is 10ms too slow, or is 10 miunutes too slow?

You need to decide what "Too slow" is and work from there.

P.P.S Think carefully before deciding to use the file system in this way, there are better alternatives (as shown in other anawers).

I would say consider using a XML file to store your 3000 pairs of data, rahter than 3000 separate files, and there are plenty of LINQ to XML you can use to access data faster.