Data processing benchmarks

We are u开发者_开发百科sing a ORM to interact with Microsoft SQL Server database and Oracle database, we do a lot of record processing and updations. I am looking into how I should benchmark my data processing capabilities.Should it be time versus data processed ?

GOALS:

• Amount of time it takes to process invalid records to valid records.(trace/timer logging I guess in code.)
• Amount of memory consumed to process a max record set (How can I do this?)

Any suggestions or guidelines welcome.

The design of a benchmark depends on what your concern is. In general, you want to be very clear about what you are trying to measure first and then work towards benchmarks after that. Presumably you have some idea of how your system behaves at the start and where your bottlenecks of concern are. If you don't have that, start by profiling, not benchmarking.

Next you want to design based on what you are worried about regarding specific bottlenecks. Build your benchmark and measure the throughput of the specific scenarios of concern.

I have built benchmarks before. Usually following these principles (some are not strictly correct but they are useful anyway):

1. Database performance is a meaningless term.
2. Application performance is a meaningless term.
3. Benchmarks measure performance of certain paths, possibly in combination with other paths.
4. The purpose of a benchmark is to get a sense of a tradeoff regarding decisions.

If you follow these you will get a benchmark that at least delivers something useful.