Best practices for testing other tests?

Starting to write unit tests (and other kinds), I realize that some tests (acceptance tests and the like) can quickly grow into quite complex software, and felt that maybe some unit tests (or at least shorter tests than the or开发者_开发技巧iginal one) to verify the longer test might be in place.

"Testing tests" might sound silly, but I wanted to know if there are people practicing this and if there is any known best practice for "Testing tests"?

The tests test the production code, and the production code should test the tests. ;-) That is to say - What you mainly want to test about a test is that it properly tests the code it's intended to test. The principal way to do this is to make the code wrong and see that the tests catch it.

There are tools for doing this automatically and measuring test coverage by seeing what mutations you can do to the code that are not caught by a test. See Mutation Testing.

If your tests are complex, you should be refactoring them and extracting some of the complexity out into test helpers. With enough of these, you might end up building something resembling a test framework, and that would deserve tests. Indeed, JUnit, RSpec, Cucumber, FitNesse, etc. are all test frameworks and have suites of tests to test the framework.

You should try to keep the tests themselves simple enough that you trust they work. But verifying by mutation testing might be worthwhile, especially as it also gives you a measure of your coverage.

Big tests have little tests upon their backs to verfy'em.
And Little tests have lesser tests, and so ad infinitum.

Loosely based on Jonathan Swift's little poem

Don't test tests. Simply assume they're correct and move on.

If the lower tests fail, of course the higher tests will fail. That's understandable. (I would advise making a note that "This test relies on test X, Y and Z passing." if it isn't intuitive by a quick overview of the higher test.)But you don't need a specific test to make sure another test works before you move on - the test suite should be running ALL tests.

I think if you have to test your tests you're doing something wrong. In my opinion tests should be really easy and simple. However, what you can do is measure how good your tests are. Maybe you want to know how much code coverage your tests have.. but that's not testing the tests as you thought, is it?

In any case, it's probably a waste of time.. :)

You've touched on a few things here. Let me try to cover them all ...

If you've written an in-house test framework, then it should have some tests around it. If you're not using the off-the-shelf frameworks and tools, then you need to know that the framework itself is working the way it is expected to. Tests will provide that.

Tests are a double-check of the code. It is better to test the code again from a different angle than to write a test of the test. I would spend time expanding my test data set or building a fuzz tester or some other such exercise to expand test coverage than spend time writing code to test the tests themselves.

You should formalize the testing process. Having a test plan (even if it is just a list of scenarios with repro steps in a wiki) is a huge step up from where most people are in their testing efforts. You should also have some form of code review for tests before/when they are checked into the code repository. This will help catch little errors like, "Oh ... you didn't realize that when x is true that it always returns five?"

When a test fails, one of the first things that a tester should check is if the test is correct. (Often because that's the first thing the developer is going to insist, that the test is wrong and their code is right.) Swallow your pride and verify the test is 100% correct, then try to figure out if there is a bug in the shipping code. You are using bug tracking software, right? When a test is broken, a bug is filed, isn't it? Sharing these test bugs helps everyone learn how to be better testers.

And finally, when all the tests are passing is when the tester should be the most vigilant! Double-check everything again and use this time to expand the test coverage in new ways, like fuzz testing, model-based testing, etc.