I'm trying to add test coverage reporting to an Android app project. The app is written in Kotlin and build with Gradle (Groovy variant) into 5 flavors.
The following Gradle plugins are used:
plugins {
id 'com.android.application'
id 'kotlin-android'
id 'kotlin-kapt'
}
I've been following a lot of different guides on integrating JaCoCo, but none of them worked for me. A minor breakthrough happened when I realized that the Android gradle plugin has (criminally underdocumented) builtin support for generating JaCoCo .exec files by adding enableUnitTestCoverage true into buildTypes { debug { ... } }.
However, I need the report in XML format for publishing the results into GitHub PRs using this Action.
I've also found that including testCoverageEnabled in the same block above adds the Gradle task create${flavor}CoverageReport which seems to do what I'm looking for
createProdMainNetDebugCoverageReport - Creates test coverage reports for the prodMainNetDebug variant.
However, running this task complains that no device is connected:
Execution failed for task ':app:connectedProdMainNetDebugAndroidTest'.
> com.android.builder.testing.api.DeviceException: No connected devices!
How can I get a code coverage report for all unit tests in XML format using the Android Gradle Plugin? Or would I be better served by configuring it more manually? How would I avoid my configuration conflicting with the existing one?
The other answer by @Fah helped be find the solution that I ended up using. Also, there are great many posts around the web describing how people have integrated JaCoCo into their projects in various ways.
The solutions are mostly similar, but all a little different because they fit into different projects. And none of them worked directly for my setup. So while the problem really isn't very complicated, it's important to understand how the pieces fit together in order to make the right adjustments for your particular project. The following is my attempt as a Gradle noob at explaining all these pieces from what I've gathered while investigating this:
Fundamentally, the solution breaks down to
Enable coverage instrumentation/collection in the unit test runner.
Define one or more Gradle tasks of type JacocoReport (which is defined by the jacoco plugin?). This task will collect the execution data produced by the test runner into a report and therefore depends on a task for actually running the tests.
Configure the task(s) with the correct locations of source, target, execution files, etc.
Add any additional JaCoCo customizations.
As mentioned in the question, the Android Gradle Plugin (AGP) already includes JaCoCo support. This means that we don't need to explicitly enable the jacoco plugin. It looks like the used version can be configured using
android {
jacoco {
version = "..."
}
}
This blog post discusses various shortcomings on AGP's JaCoCo support and proposes a Gradle plugin (in Kotlin) to address them.
AGP supports this directly by setting enableUnitTestCoverage on the appropriate build type:
android {
...
defaultConfig {
...
debug {
...
enableUnitTestCoverage true
}
}
There also is the testCoverageEnabled (deprecated combination of enableUnitTestCoverage and enableAndroidTestCoverage) option which additionally creates tasks that we can't use when running in CI without an actual device as described in the question.
Gradle tasks are defined with the tasks.create method. A task has a name, a type, and a set of other tasks that it depends on. In this case this means something like
tasks.create(name: "unitTestCoverageReport", type: JacocoReport, dependsOn: "testLocalDebugUnitTest") {
// ... task implementation ...
}
where testLocalDebugUnitTest is an existing task that runs the unit tests with coverage collection enabled. Use ./gradlew tasks | grep -i test to list all available tasks related to testing.
The task can be defined in the gradle.build file of the module that needs it (what I did), or it can be defined in a separate file to be applied from the toplevel module as in @fah's answer or imported explicitly as in some of the posts listed below.
By virtue of this being written in a full programming language, there are many possibilities for setting up abstractions around this construct in whatever way makes sense for the project at hand:
This example shows how JaCoCo may be integrated into a monorepo-like project, i.e. a project with multiple Gradle modules. It defines a generic JaCoCo Gradle module that can be imported into the individual build.gradle files to add a debugCoverage task for each of them (assuming that they have a variant "debug"). It then adds a dummy module jacoco with a single task allDebugCoverage that collects the coverage data produced by debugCoverage from all the other modules (assuming that they have been run first). The JaCoCo Report Aggregation Plugin looks like a possible alternative solution.
This example creates a task for each variant.
This example creates it for each flavor.
With the task unitTestCoverageReport being defined, it may be run using ./gradlew unitTestCoverageReport. Dependent tasks will be automatically be executed (transitively) first.
As mentioned above, the task implementation sets up the configuration of JaCoCo report generator. As also seen in @Fah's answer, the relevant fields are:
reports: block for configuring which kinds of reports to generate.
sourceDirectories.from: The location of the original source code files. Optional, but allows the reporter to map bytecode instructions back to the original sources in order to include them into the report with coverage annotations.
classDirectories.from: The compiled class files without instrumentation. Execution data references instructions in these classes, so they're necessary for locating the actual instructions.
executionData.from: The location of "execution data", i.e. the coverage data collected by the unit test runner when running the tests against classes instrumented by the JaCoCo agent.
So while one just has to set these paths correctly, this can be challenging when the build process is poorly documented and you don't have good understanding of the underlying structure. In my case it involved a combination of guessing, searching the build directory for files with specific extensions, and reading other examples. I have for example not found any official documentation explaining that compiled Kotlin classes go into ${buildDir}/tmp/kotlin-classes.
Despite being named "directories", all guides I've seen pass the entire directory tree using the fileTree function with a bunch of excludes. For my Kotlin Android project, the following configuration suffices:
tasks.create(name: "unitTestCoverageReport", type: JacocoReport, dependsOn: "testLocalDebugUnitTest") {
group = "Verification" // existing group containing tasks for generating linting reports etc.
description = "Generate Jacoco coverage reports for the 'local' debug build."
reports {
// human readable (written into './build/reports/jacoco/unitTestCoverageReport/html')
html.enabled = true
// CI-readable (written into './build/reports/jacoco/unitTestCoverageReport/unitTestCoverageReport.xml')
xml.enabled = true
}
// Execution data generated when running the tests against classes instrumented by the JaCoCo agent.
// This is enabled with 'enableUnitTestCoverage' in the 'debug' build type.
executionData.from = "${project.buildDir}/outputs/unit_test_code_coverage/localDebugUnitTest/testLocalDebugUnitTest.exec"
// Compiled Kotlin class files are written into build-variant-specific subdirectories of 'build/tmp/kotlin-classes'.
classDirectories.from = "${project.buildDir}/tmp/kotlin-classes/localDebug"
// To produce an accurate report, the bytecode is mapped back to the original source code.
sourceDirectories.from = "${project.projectDir}/src/main/java"
}
I think the XML report is mostly useful for further processing like uploading the report to coveralls.
Some guides wrap the task definition in project.afterEvaluate. I didn't find a need for this and don't know enough about Gradle to understand what difference it makes.
To collect coverage data for tests executed by for example RobolectricTestRunner, the following snippet must be added as well:
tasks.withType(Test) {
jacoco.includeNoLocationClasses true
jacoco.excludes = ['jdk.internal.*']
}
This post suggests using the testOptions block to set configurations like these.
Finally, the jacoco plugin also includes a configuration jacocoTestCoverageVerification for failing the build if certain coverage metrics aren't met.
The following example workflow runs the Gradle task and uploads the generated HTML report as a workflow artifact using madrapps/jacoco-report:
name: Build and test with coverage
on: ...
env:
java_version: "17"
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up JDK
uses: actions/setup-java@v3
with:
java-version: "${{env.java_version}}"
distribution: temurin
cache: gradle # caches dependencies (https://github.com/actions/setup-java#caching-packages-dependencies)
- name: Build project and run unit tests
run: ./gradlew test unitTestCoverageReport --no-daemon
- name: Add coverage comment to PR
uses: madrapps/[email protected]
with:
paths: "${{github.workspace}}/app/build/reports/jacoco/unitTestCoverageReport/unitTestCoverageReport.xml"
token: "${{secrets.GITHUB_TOKEN}}"
- name: Store HTML coverage report
uses: actions/upload-artifact@v3
with:
name: coverage-report
path: "${{github.workspace}}/app/build/reports/jacoco/unitTestCoverageReport/html/"