Code coverage for Android (calabash-android BDD)

I'm testing my android application using calabash-android which provides it's own "test project", with a script that renames it's package to reflect the app under test, and then uses an InstrumentationTestRunner subclass:

adb shell am instrument -w -e class sh.calaba.instrumentationbackend.InstrumentationBackend #{ENV['TEST_PACKAGE_NAME']}/sh.calaba.instrumentationbackend.CalabashInstrumentationTestRunner

I'll accept any answer that allows me to generate a code coverage report similar to Emma or Cobertura for an Android app, with data collected while being tested in calabash-android.

In attempt to get Emma working I have...

Attempted to follow these instructions to get Maven to build my project (because it's been too long since I used ant). A coverage.em file is generated in target/emma
Modified the calabash-android script, adding "-e coverage true"
When I run calabash-android, I eventually see "Generated coverage data to /data/data/my.project/files/coverage.ec"
adb -e pull /data/data/my.project/files/coverage.ec coverage.ec

...so now I should be able to run:

java -cp $ANDROID_HOME/tools/lib/emma.jar emma report -r html -in target/emma/coverage.em,coverage.ec

but I get an error:

EMMA: processing input files...
java.io.UTFDataFormatException: malformed input around byte 107

...So I assume that there's something wrong with the android maven plugin, and I'm trying to figure out how to generate the coverage.em file. I've ran "android update project -p ." and "ant emma" and "ant emma debug", but I can't find coverage.em anywhere...

...The generated build.xml file seems to imply that the generation of the coverage.em file is only generated when you run "ant emma test", but I don't think that's going to work because the test app is controlled by calabash-android.

In attempt to get Cobertura working I have...

Googled various forms of "cobertura android", but it doesn't seem as if anyone has had any luck.
Attempted to configure a cobertura profile in my Maven pom file to instrument the classes, but (in Maven 3) I get
- a whole heap of warnings about log4j and ant having "InnerClasses" attributes and that I should recompile them from source
- an error that com.jayway.maven.plugins.android.generation2:android-maven-plugin:3.3.0:dex "ANDROID-040-001: Could not execute: Command = /bin/sh -c -cd /path/to/myproject && java -jar $ANDOID_HOME/platform-tools/lib/dx.jar --dex ..."

EXCEPTION FROM SIMULATION:
local variable type mismatch: attempt to set or access a value of type 
java.lang.Class using a local variable of type java.lang.reflect.Type[].  
This is symptomatic of .class transformation tools that ignore local variable information.

... this is probably why nobody's been able to get cobertura working on Android?

Solution

The problem is that maven-android-plugin uses version 2.1.5320 of emma, whereas the Android tools use version 2.0.5312. As discussed here, those two versions are incompatible.

The fix is to only use a single version across the two tools. I've been able to get it to work by cloning the maven-android-plugin repo, setting the emma dependency version back to 2.0.5312, and installing it to my local repository. Make sure the emma dependency in your under-test project is also correct, and then you should be able to generate a coverage.

The alternative is to make sure all of the tools are using the newest version. I haven't tested it, but it might work if you do the report generation from maven is the version will be the same then. You could also download the newest version of emma and generate the report using the jar from that package.