How to store data when there is no schema?

I'm trying to figure out what is the right choice of data storage in a project I'm starting up right now.

I want to store data that is the output result of powershell scripts. This means that an administrator of my app will be able to write a powershell script that will execute on a number of hosts and they will post the results back to a data store. I then want to query that store in a flexible manner.

Let me clarify. The data that 开发者_JAVA技巧comes back from the powershell job is not a proper object but a key/value collection of properties of objects. So there is no real object to serialize.

Let's say I tell 100 hosts over a WCF service to execute the two powershell commands Get-Service and Get-Process and they will then post back the results to my data store. I don't know the schema of this data beforehand.

The point is not PowerShell nor WCF, but how would you store data that at the time of storing the schema is not known. And querys will be created manually via some GUI afterwards based on the data that has been stored.

Afterwards I would like to be able execute a query like "Get a list of all hosts that have service X running and process Y running" ?

I'm looking into at nosql databases as an alternative to relational DBs but not sure what is best.

Thankful for any input. /Linus

If storing the data as XML to an RDBMS doesn't make sense to you (btw, why doesn't it?), then there are several NoSQL DBs that would probably be good options because they're schema-less.

The ones I can recommend that you look at (based on personal experience, there are many others that could be relevant) are CouchDB and Riak. Both provide a disk-bound key-value datastore where you store your values as JSON, w/o pre-defining a schema. In both cases it is possible to query the data through a RESTful interface using Javascript.

The choice should depend on the amount of data that you expect:

• Riak is designed to run on multiple nodes, and queries are handled through MapReduce so that processing is distributed between those nodes, enabling relatively fast data retrieval for ad-hoc queries. If you have lots of data - millions of records that you must run ad-hoc queries, choose this. You'll 'pay' with the added complexity of managing a cluster, though I can attest that Riak makes it relatively painless.
• CouchDB is designed to run on a single node. Replication is possible (and easy) but queries run against a single server. It has materialized indices, so queries against existing indices run fast. Ad-hoc queries require a full "table scan" though, and could take minutes on large datasets. OTOH, it has the benefit of a nice browser-based user interface that Riak lacks in the free version.

I'd recommend trying Couch out first - it's very easy to set up and start playing with - and see whether it solves your problem. If it doesn't, then go for Riak.

If you want to store data of which you do not know the structure during design time, you have a few options.

Amongst the options are:

Store data as xml (in DB or files).

Create schema dynamically to match dynamic data's structure.

Create a generic structured schema, where all classes map to the same table, and all properties are dynamically attached properties.

E.g. (Generic class structure)

GenericClass
{
    GenericProperty[] SimpleProperties;
    Dictionary[string, GenericClass] ComplexProperties;
}

GenericProperty
{
    String Name;
}

StringProperty: GenericProperty
{
    String Value;
}

IntegerProperty: GenericProperty
{
    Integer Value;
}

Using table-per-type on these classes should give you generic tables.