The problem of metrics/report generation

We store a lot of metrics from our service (approx 80 million events). We have to generate reports based on the data.

My question is rather general, What tools do u开发者_高级运维se for your metrics/reporting need? Is there anything recommended?

We use Apache to write the log files, back-process to add it into the DB, and a daily MySql script to generate reports.

Many thanks,

SenSage. Expensive and worth it.

It really depends on the type of reports you are generating.

If you are handling things at the level of mean/count/sum/deviance/min/max of a population, there's a variety of tools that can handle what you want. Open source tools like Graphite/Statsd (using Statsd can also give you things like the quantile of a population) work well. There are commercial tools like my company's Instrumental product which provide some of the same features, as well as niceties like a good UI and well documented API.

Depending on the nature of your data, there are other services that offer more specific report types like cohort analysis (see Mixpanel ). Services like Splunk can let you create some great ad hoc reports of live data as well, generated from your application logs and other data sinks.