Automatic / templated generation of test vectors in C++

I want to find a good way to generate test vectors automatically. By way of example, I am testing an audio processing module by calling a function that exercises the module-under-test with the specified test vector and in doing so makes various checks for proper operation and correctness of module output.

void runTest(const char *source, double gain, int level);

The test vector is the triplet of source, gain and level. Here is the multidimensional space I want to test against:

const char *sources[] = {"guitar.mp3", "vocals.mp3", "drums.mp3"};
double gains[] = {1., 10., 100.};
int levels[] = {1, 2, 3, 4};

Values can have other properties, for example if vocals.mp3 has a dynamic rage of 2, guitar 5 and drums 10, we could conceive a representation like:

int dynamicRange(const char *source);

I want to be able to configure various test runs. For example, I want to be able to run:

// all permutations (total 36 vectors)
runTest("guitar.mp3", 1., 1);
runTest("guitar.mp3", 1., 2);
runTest("guitar.mp3", 1., 3);
runTest("guitar.mp3", 1., 4);
runTest("guitar.mp3", 1., 1);
runTest("guitar.mp3", 10., 2);
runTest("guitar.mp3", 10., 3);
// ...

// corner cases (according to dynamicRange)
runTest("vocals.mp3", 1., 1);
runTest("vocals.mp3", 1., 4);
runTest("vocals.mp3", 100., 1);
runTest("vocals.mp3", 100., 4);
runTest("drums.mp3", 1., 1);
runTest("drums.mp3", 1., 4);
runTest("drums.mp3", 100., 1);
runTest("drums.mp3", 100., 4);

// sparse / minimal tests touching every value for each parameter
runTest("guitar.mp3", 1., 1);  
runTest("vocals.mp3", 10., 2);  
runTest("drums.mp3", 100., 3);  
runTest("guitar.mp3", 1., 4);  

// quick test
runTest("guitar.mp3", 1., 1);

I want create the above code without lots of copy and paste either dynamically or using my compiler to do the legwork, for example:

// syntax tentative here, could be class/template instantiations
allPermutations(runTest, sources, gains, levels);
cornerCases(runTest, lookup(sources, dynamicRange), gains, levels);
minimal(runTest, sources, gains, levels);
quick(runTest, sources, gains, levels);

The above looks like dynamic C but my language is C++ and I am expecting to use templates and some combination of dynamic and static techniques. Perhaps even metaprogramming.

Combinations and variations would also be interesting. For example, I might want to use only the shortest input file. Or I might want to run all sources with corner-cases for gain and level. Or gain could also be a continuous range开发者_开发技巧 1 to 100 but let's keep things discrete for now.

Before I start designing types, templates, representation, etc. I wondered if this is a problem that has been solved before or, if not, would any existing libraries, e.g. Boost MPL, be useful?

I think it would be useful if you introduce yourself to concept of All-pairs testing, and have a quick check for QuickCheck (it is the Haskell test framework which generates test cases randomly according to the given spec, and then checks that some properties are hold; there exists C++ version of it).

Regarding Boost.MPL in particular, I don't think it would help you for this task at all: you are not dealing with list of types here, are you.

My another advise on your upcoming design: don't overgeneralize. Before you start with types, templates, etc. implement 3 (three) reasonably different implementations, and then generalize what you already have at hand.

It was very tempting to think a bit about this very programmer-friendly task :)

Here I came with dynamic solution using boost::any as the medium to store "erased" types in. More static solution would use probably Boost.Tuple and Boost.Fusion/Boost.MPL indeed, but I'm not sure it is worth the trouble.

The code is prototype-quality, and for sure you are not going to use it as it is. But at least it can give you direction.

So the mini-framework:

typedef boost::option<boost::any> OptionalValue;
OptionalValue const no_value;

// represents each dimension from your multi-dimensional solution
struct Emitter
{
    virtual ~Emitter() { }

    // should return no_value to indicate that emitting finished
    virtual OptionalValue emit() = 0;
};
typedef boost::shared_ptr<Emitter> EmitterPtr;

// generates test vectors according to passed emitters and run test function on each
class Generator
{
public:

    void add_emitter(EmitterPtr p) { emitters.push_back(p); }

    // here f is callback called for each test vector
    // could call test, or could store test vector in some container
    template <class F>
    void run(F f)
    {
        std::vector<boost::any> v;
        generate(v, 0, f);
    }

private:

    template <class F>
    void generate(vector<boost::any>& v, size_t i, F f)
    {
        if (i == emitters.size())
        {
            f(v);
        }

        EmitterPtr e = emitters[i];
        for (OptionalValue val = e->emit(); val; )
        {
            v.push_back(*val);
            generate(v, i + 1, f);
            v.pop_back();
        }
    }

private:
    std::vector<EmitterPtr> emitters;
};

Some concrete emitters:

// emits all values from given range
template <class FwdIt>
struct EmitAll : Emitter
{
    EmitAll(FwdIt begin, FwdIt end) : current(begin), end(end) { }
    OptionalValue emit() { return current == end ? no_value : *(current++); }

    FwdIt current;
    FwdIt const end;
};

// emits first value from given range, and finshes work
template <class FwdIt>
struct EmitFirst : Emitter
{
    EmitFirst(FwdIt begin, FwdIt) : current(begin), n(0) { }
    OptionalValue emit() { return n++ == 0 ? *current : no_value; }

    FwdIt current;
    size_t n;
};

// emits only values satisfied predicate P
template <class FwdIt, class P>
struct EmitFiltered
{
    EmitFiltered(FwdIt begin, FwdIt end) : current(begin), end(end) { }
    OptionalValue emit()
    {
        P const p;
        while (current != end)
        {
            if (!p(current)) continue;
            return *(current++);
        }
        return no_value;
    }

    FwdIt current;
    FwdIt const end;
};

// helpers for automatic types' deducing
template <class FwdIt>
EmitterPtr make_emit_all(FwdIt b, Fwd e) { return new EmitAll<FwdIt>(b, e); }

template <class FwdIt>
EmitterPtr make_emit_first(FwdIt b, Fwd e) { return EmitFirst<FwdIt>(b, e); }

template <class FwdIt>
EmitterPtr make_emit_filtered(FwdIt b, Fwd e, P p) { return EmitFiltered<FwdIt, P>(b, e, p); }

Adapter for runTest:

struct Run
{
    void operator()(const std::vector<boost::any>& v)
    {
        assert v.size() == 3;
        runTest(boost::any_cast<std::string>(v[0]),
                boost::any_cast<double>     (v[1]),
                boost::any_cast<int>        (v[2]));
    }
};

Finally usage:

Generator all_permutations;
all_permutations.add_emitter(make_emit_all(sources, sources + 3));
all_permutations.add_emitter(make_emit_all(gains,   gains + 3));
all_permutations.add_emitter(make_emit_all(levels,  levels + 4));

Generator quick;
quick.add_emitter(make_emit_first(sources, sources + 3));
quick.add_emitter(make_emit_first(gains,   gains + 3));
quick.add_emitter(make_emit_first(levels,  levels + 4));

Generator corner_cases;
corner_cases.add_emitter(make_emit_all(sources, sources + 3));
corner_cases.add_emitter(make_emit_filtered(gains, gains + 3, LookupDynamicRange));
corner_cases.add_emitter(make_emit_all(levels,  levels + 4));

Run r;
all_permutations.run(r);
quick.run(r);
corner_cases(r);

Implementing all-pairs beast (for 'minimal' guy) is left to you to implement %)

You might be interested in Template2Code framework. It is especially designed for solving your problem. The comprehensive documentation is here. According to the documentation you should create a *.t2c file of the following structure to generate a complete set of test vectors:

<BLOCK>
    ...
    <DEFINE>
        #define SOURCE <%0%>
        #define GAIN <%1%>
        #define LEVEL <%2%>
    </DEFINE>
    <CODE>
        runTest(SOURCES, GAINS, LEVELS);
    </CODE>
    <VALUES>
        SET("guitar.mp3"; "vocals.mp3"; "drums.mp3")
        SET(1.; 10.; 100.)
        SET(1; 2; 3; 4)
    </VALUES>
    ...
</BLOCK>

This technology was used by The Linux Foundation and ISPRAS to create "normal"-quality tests for libstdcxx, glib, gtk, fontconfig, freetype and other libraries.