How should I deal with failing tests for bugs that will not be fixed

I have a complex set of integration tests that uses Perl's WWW::Mechanize to drive a web app and check the results based on specific combinations of data. There are over 20 subroutines that make up the logic of the tests, loop through data, etc. Each test runs several of the test subroutines on a different dataset.

The web app is not perfect, so sometimes bugs开发者_如何学C cause the tests to fail with very specific combinations of data. But these combinations are rare enough that our team will not bother to fix the bugs for a long time; building many other new features takes priority.

So what should I do with the failing tests? It's just a few tests out of several dozen per combination of data. 1) I can't let it fail because then the whole test suite would fail. 2) If we comment them out, that means we miss out on making that test for all the other datasets. 3) I could add a flag in the specific dataset that fails, and have the test not run if that flag is set, but then I'm passing extra flags all over the place in my test subroutines.

What's the cleanest and easiest way to do this? Or are clean and easy mutually exclusive?

That's what TODO is for.

With a todo block, the tests inside are expected to fail. Test::More will run the tests normally, but print out special flags indicating they are "todo". Test::Harness will interpret failures as being ok. Should anything succeed, it will report it as an unexpected success. You then know the thing you had todo is done and can remove the TODO flag.

The nice part about todo tests, as opposed to simply commenting out a block of tests, is it's like having a programmatic todo list. You know how much work is left to be done, you're aware of what bugs there are, and you'll know immediately when they're fixed.

Once a todo test starts succeeding, simply move it outside the block. When the block is empty, delete it.

I see two major options

1. disable the test (commenting it out), with a reference to your bugtracking system (i.e. a bug ig), possibly keeping a note in the bug as well that there is a test ready for this bug

2. move the failing tests in a seperate test suite. You could even reverse the failing assertion so you can run the suite and while it is green the bug is still there and if it becomes red either the bug is gone or something else is fishy. Of course a link to the bugtracking system and bag is still a good thing to have.

If you actually use Test::More in conjunction with WWW::Mechanize, case closed (see comment from @daxim). If not, think of a similar approach:

# In your testing module
our $TODO;
# ...
if (defined $TODO) {
    # only print warnings
};

# in a test script
local $My::Test::TODO = "This bug is delayed until iteration 42";