Selenium IDE - always fail on any 500 error

Is there an easy way to tell Selenium IDE that any action that results in a http 500 response means the test failed?

I have tests that are 75 page requests long. Sometimes, I get a crash and burn somewhere in the middle, but the tests come back green.

Taking a look at selenium-api.js, I saw that there is a parameter ignoreResponseCode in the signature of the doOpen method in selenium-api.js :

Selenium.prototype.doOpen = function(url, ignoreResponseCode) {

This parameter is used by the browserbot object :

if (!((self.xhrResponseCode >= 200 && self.xhrResponseCode <= 399) || self.xhrResponseCode == 0)) {
    // TODO: for IE status like: 12002, 12007, ... provide corresponding statusText messages also.
    LOG.error("XHR failed with message " + self.xhrStatusText);
    e = "XHR ERROR: URL = " + self.xhrOpenLocation + " Response_Code = " + self.xhrResponseCode + " Error_Message = " + self.xhrStatusText;
    self.abortXhr = false;
    self.isXhrSent = false;
    self.isXhrDone = false;
    self.xhrResponseCode = null;
    self.xhrStatusText = null;
    throw new SeleniumError(e);
}

I've tried calling the open function from selenium IDE with value = false and this results in an error (test failed).

My PHP test page was :

<?php
    header('HTTP/1.1 500 Simulated 500 error');
?>

And this results in :

For me, this solves the problem of checking HTTP response status.

Make a JavaScript file called "user-extensions.js" and add it to the Selenium-IDE under Options > Options. If you are running Selenium RC, pass it into the parameter when starting up your server in the jar command. There should be a user extensions javascript file attribute.

Then close and restart Selenium-IDE. The User-Extensions file is cached when the IDE starts up.

Add this code to your Selenium user-extensions.js file to make a custom command called "AssertLocationPart". As you know "assertLocation" and "storeLocation" are standard commands. I tried to reduce the extra line of code to storeLocation just by getting the href in the custom function. I wasn't able to get the doAssertValue command to work. I'll have to post my own question for that. That's why it's commented out. For now, just use "this.doStore" instead. And add an extra line to your script after your custom AssertLocationPart command. Since we're not actually doing an assertion in the custom function/command, we should call it "storeLocationPart" (function would be named "doStoreLocationPart"), not "assertLocationPart" (function would be named "doAssertLocationPart"), and just pass in the first parameter. But if you can get the doAssert* to work, please let me know. I'll mess with it another day since I need to do this same thing for work.

Selenium.prototype.doAssertLocationPart = function(partName,assertTo) { 

    var uri = selenium.browserbot.getCurrentWindow().document.location.href;
    //alert("URI = " + uri);

    var partValue = parseUri(uri,partName);
    //alert("Part '" + partName + "' = " + partValue);

    //this.doAssertValue(partValue,assertTo);
    this.doStore(partValue,"var_"+partName);
};


// Slightly modified function based on author's original:
// http://badassery.blogspot.com/2007/02/parseuri-split-urls-in-javascript.html
//
// parseUri JS v0.1, by Steven Levithan (http://badassery.blogspot.com)
// Splits any well-formed URI into the following parts (all are optional):
//
// - source (since the exec() method returns backreference 0 [i.e., the entire match] as key 0, we might as well use it)
// - protocol (scheme)
// - authority (includes both the domain and port)
//     - domain (part of the authority; can be an IP address)
//     - port (part of the authority)
// - path (includes both the directory path and filename)
//     - directoryPath (part of the path; supports directories with periods, and without a trailing backslash)
//     - fileName (part of the path)
// - query (does not include the leading question mark)
// - anchor (fragment)
//
function parseUri(sourceUri,partName){
    var uriPartNames = ["source","protocol","authority","domain","port","path","directoryPath","fileName","query","anchor"];
    var uriParts = new RegExp("^(?:([^:/?#.]+):)?(?://)?(([^:/?#]*)(?::(\\d*))?)?((/(?:[^?#](?![^?#/]*\\.[^?#/.]+(?:[\\?#]|$)))*/?)?([^?#/]*))?(?:\\?([^#]*))?(?:#(.*))?").exec(sourceUri);
    var uri = {};

    for(var i = 0; i < 10; i++){
        uri[uriPartNames[i]] = (uriParts[i] ? uriParts[i] : "");
        if (uriPartNames[i] == partName) {
            return uri[uriPartNames[i]];  // line added by MacGyver
        }
    }

    // Always end directoryPath with a trailing backslash if a path was present in the source URI
    // Note that a trailing backslash is NOT automatically inserted within or appended to the "path" key
    if(uri.directoryPath.length > 0){
        uri.directoryPath = uri.directoryPath.replace(/\/?$/, "/");
        if (partName == "directoryPath") {
            return uri.directoryPath;  // line added by MacGyver
        }
    }

    return uri;
}

Then add this to your web.config file and make sure customErrors is turned off. Since you have a 500 error, it will redirect the user to the default page. Feel free to add a custom page for a 500 HTTP status code if you want to be specific in your Selenium scripts.

<customErrors mode="On" defaultRedirect="/ErrorHandler.aspx">
    <error statusCode="401" redirect="/AccessDenied.aspx" />
    <error statusCode="403" redirect="/AccessDenied.aspx" />
    <error statusCode="404" redirect="/PageNotFound.aspx" />
</customErrors>

This is what your commands will look like in the IDE:

Make sure you're on this page (or something similar) before running the script:

https://localhost/ErrorHandler.aspx?aspxerrorpath=/path/pathyouweretryingtoviewinwebapp.aspx

Log shows that it passed!

• [info] Executing: |storeLocation | var_URI | |
• [info] Executing: |echo | ${var_URI} | |
• [info] echo: https://localhost/ErrorHandler.aspx?aspxerrorpath=//path/pathyouweretryingtoviewinwebapp.aspx
• [info] Executing: |assertLocationPart | fileName | ErrorHandler.aspx |
• [info] Executing: |assertExpression | ${var_fileName} | ErrorHandler.aspx |

Using the error handler from my previous answer:

Command: assertLocation
Target: regexp:^(https://localhost/ErrorHandler.aspx).*$

Or (per your comment) it's inverse if you don't have error handling turned on, use AssertNotLocation. This may require more work on the person writing the scripts. You'd have to keep track of all pages.

More on pattern matching:

http://seleniumhq.org/docs/02_selenium_ide.html#matching-text-patterns

http://www.codediesel.com/testing/selenium-ide-pattern-matching/