PHPUnit - Automatically retry failed tests X times?

I have a complex set of PHPUnit tests, some of which involve connecting to servers around the world that, for whatever reason, timeout sometimes.

Rather than having the test fail when the server times out, I'd like to simply retry that test one or more times before actually marking it as failed.

Now, I understand that this may not be the best way to handle my situation at hand. One better solution would be to fix the servers. But, this is out of my control right now.

So, what I'd really like, is a way t开发者_JAVA百科o tell PHPUnit to re-test each failing testcase X times, and only mark it as failed if it failed every time.

Any ideas?

Edit: Many of you have responded with helpful suggestions that I not do this. I understand, thank you. However, specifically what I am trying to do is create a test suite that tests the operation of the full system, including the remote servers. I understand the concept of testing certain parts of my code with "mock" responses from the outside...but I also sleep better at night if part of my tests test the "full stack".

As PHPUnit doesn't support this behavior out of the box, you'll need to code the loop yourself. Instead of doing it in every test that requires it, create a custom test case base class (if you haven't already) that extends PHPUnit_Framework_TestCase and provides the feature.

You can either get fancy and override testBare() to check for an annotation such as @retry 5, loop that number of times, calling parent::testBare(), and swallow all exceptions (or a subset) except the last.

public function runBare() {
    // I'll leave this part to you. PHPUnit supplies methods for parsing annotations.
    $retryCount = $this->getNumberOfRetries();
    for ($i = 0; $i < $retryCount; $i++) {
        try {
            parent::runBare();
            return;
        }
        catch (Exception $e) {
            // last one thrown below
        }
    }
    if ($e) {
        throw $e;
    }
}

Or you can create a similar helper method that takes the number of retries and a closure/callable as parameters and call it from each test that needs it.

public function retryTest($count, $test) {
    // just like above without checking the annotation
    ...
        $test();
    ...
}

public function testLogin() {
    $this->retryTest(5, function() {
        $service = new LoginService();
        ...
    });
}

Not exactly an answer to your question, but I'll say it anyway: Your tests should never include remote resources (especially when they are completely out of your hand (unlike local mirrors)). You should encapsulate your connections into separate classes (e.g. Connection) and within your tests you mock those objects and work with static responses, that your remote hosts would return.

Rather than connecting to live servers, shouldn't you be using mock objects and fixtures so that responses from elsewhere don't have an effect on your tests?

You could possibly use dependency injection to use a specific HTTP client that will return the data and response code you tell it to (depending on how your code is written). Ideally, your unit tests should be independed of outside influences; you should be in control of what you're testing, and forcing a 404 or 500 error, for example, should be a separate part of your tests.

Rather than trying to hack around non-deterministic tests, you would be better served by looking at whether you can alter your code to enable mocking and test fixtures.

Aside from that, which you may already know of course, I'm afraid I don't know of a way to tell PHPUnit to allow a test to fail. It seems completely contrary to what the tool should be doing.

I don't think there is a support for that, someone can prove me wrong, but I'd be surprised in that case.

I think what you can do is instead of asserting right away in the test methods to be retrieved is looping for X times and break out on success, outside the loop the result will be asserted.

This simple way, that you'd already thought about, has the disadvantage to add some more code to each test method. If you've many of them it adds to maintenance burden.

Otherwise, if you want to automate more, you could implement a PHPUnit_Framework_TestListener, keep a count of failed tests in associative array and compare with the test runs. Not sure how feasible is this route, but you could give a try.

You should be able to create a DB connection assertion and implement that test in your other tests "as" your DB connection. Inside that test you could try as many times as you need and return false after X tries.