Failing Android JUnit tests, not breaking my Ant script like I expect?

Failing JUnit tests, not breaking my Ant script like I expect?

My continuous integration server runs an Ant script, which calls something like: /tests/ant run-tests

My JUnit tests run, but with errors: run-tests: [echo] run-tests-helper. [echo] Running tests ... [exec] [exec] com.zedray.stuff.FooBarTest:.... [exec] com.zedray.stuff.FooBarTest:.....INSTRUMENTATION_RESULT: shortMsg=Some error in your code. [exec] INSTRUMENTATION_RESULT: longMsg=java.security.InvalidParameterException: Some error in your code [exec] INSTRUMENTATION_CODE: 0

The e开发者_开发问答rrors are OK, but my build script keeps going (eventually publishing my broken app to my testers - bad!). What I would expect is for the instrimentaiton to throw a build error, so my continuous integration server (TeamCity in this case) realises that something has gone wrong and reports a broken build. The "failonerror" is already set in the relevant macrodef, so I'm not sure what else I can do?

Running tests ...

Any ideas/suggestions on how to fix this?

Regards Mark

I did it another way, because I am using the ant test target that is in the Android build.xml file. This target prints to the standard out, so I captured stndout into a file then queried the file, using this result to fail my task.

 <target name="run-acceptance-tests" depends="clean, debug, install" >

    <property name="log.file" value="acceptance_tests_standard_out.txt" />
    <!-- because we don't have control over the 'test' target (to check for passes an fails) this prints to standard out
         we capture standard out into a file and query this to see if we have any test failures, using this to pass/fail our task -->
    <record name="${log.file}" action="start" />
    <antcall target="test" />
    <record name="${log.file}" action="stop" />

    <!-- do other stuff -->

    <loadfile property="tests.output" srcFile="${log.file}" />

    <echo>Checking for failures</echo>
    <fail message="acceptance tests failed!" >
        <condition>
            <contains string="${tests.output}" substring="FAILURES" />
        </condition>
    </fail>

    <echo>acceptance tests passed!</echo>
</target>

I had the same problem, and I ened up customize the "run-tests" target in my own build.xml like this, and there is no need to change the original android sdk test_rules.xml

<target name="run-tests" depends="-install-tested-project, install"
            description="Runs tests from the package defined in test.package property">
    <echo>Running tests ...</echo>
    <exec executable="${adb}" failonerror="true" outputproperty="tests.output">
        <arg value="shell" />
        <arg value="am" />
        <arg value="instrument" />
        <arg value="-w" />
        <arg value="-e" />
        <arg value="coverage" />
        <arg value="@{emma.enabled}" />
        <arg value="${manifest.package}/${test.runner}" />
    </exec>
    <echo message="${tests.output}"/>
    <fail message="Tests failed!!!">
         <condition>
             <contains string="${tests.output}" substring="FAILURES" />
         </condition>
    </fail>
</target>

Also was looking for some kind of standard solution for this. I wonder how do android guys develop, or they dont use teamcity and continuous integration? heard hudson has some plugin for android but I dont like hudson. anyway here is quick and dirty solution

replace contents in android-sdk-windows\tools\ant\test_rules.xml with:

        <attribute name="emma.enabled" default="false" />
    <element name="extra-instrument-args" optional="yes" />
    <sequential>
        <echo>Running tests ...</echo>
        <exec executable="${adb}" failonerror="true" outputproperty="tests.output">
            <arg line="${adb.device.arg}" />
            <arg value="shell" />
            <arg value="am" />
               <arg value="instrument" />
            <arg value="-w" />
            <arg value="-e" />
               <arg value="coverage" />
               <arg value="@{emma.enabled}" />
            <extra-instrument-args />
            <arg value="${manifest.package}/${test.runner}" />
    </exec>
    <echo message="${tests.output}"/>
    <fail message="Tests failed!!!">
            <condition>
                    <contains string="${tests.output}" substring="FAILURES" />
            </condition>
    </fail>
    </sequential>

there are two drawbacks 1) you dont see test output while tests are running until they failed (and the output is crippled somewhow) 2) its better to override this macro in your project

another option would of course be to ditch Ant in favor of Maven or Gradle. Both have Android plug-ins that properly fail the build when there are test failures.

Maven: http://code.google.com/p/maven-android-plugin/

Gradle: http://code.google.com/p/gradle-android-plugin/

running instrumentation tests has just been added to the Gradle Android plug-in, and is waiting to be merged back into the master repository, so there should be another release soon.

The ant JUnit task defaults to running all the tests. There are two solutions to this.

Easiest solution is to set the haltonerror property to true and the build will fail at the first test failure.

Slightly more involved (and my preference) is to set the failureProperty so that all the tests run. This lets you know how many tests fail instead of only the first test that fails. This requires more ant work because you need to add a line after your junit tests like this:

<fail message="tests failed" if="failureProperty"/>