Search code examples
androidofflinepowermockjacocoinstrumentation

Powermock Jacoco Gradle 0% Coverage For Android Project

We have an Android project, and we're using Powermock for some of our test cases and Jacoco for coverage report. We noticed that some our classes are returning as 0% coverage although they are indeed covered. We also observed the message below for affected classes.

"Classes ... do no match with execution data."

A few searches online show that Powermock and Jacoco don't play well and that Offline Instrumentation is a possible workaround.

Has anyone used gradle Offline Instrumentation script for android projects before?

Solution

  • In hindsight, I guess this can be solved with enough android experience and online perusing. However, I was (and still am) relatively new to Android, gradle, and groovy when this fell on my lap, so I'm writing this for the next me :-D

    WHAT IS HAPPENING IN A NUTSHELL (excerpt from a jacoco forum)

    • Source file is compiled into non-instrumented class file
    • Non-instrumented class file is instrumented (either pre-instrumented offline, or automatically at runtime by Java agent)
    • Execution of instrumented classes collected into exec file
    • report decorates source files with information obtained from analysis of exec file and original non-instrumented class files
    • Message "Classes ... do no match with execution data." during generation of report means that class files used for generation of report are not the same as classes prior to instrumentation.

    SOLUTION

    The Jacoco Offline Instrumentation page provides the main steps that should occur for offline instrumentation in this excerpt:

    For such scenarios class files can be pre-instrumented with JaCoCo, for example with the instrument Ant task. At runtime the pre-instrumented classes needs be on the classpath instead of the original classes. In addition jacocoagent.jar must be put on the classpath.

    The script below does exactly that:

        apply plugin: 'jacoco'

configurations {
    jacocoAnt
    jacocoRuntime
}

jacoco {
    toolVersion = "0.8.1"
}

def offline_instrumented_outputDir = "$buildDir.path/intermediates/classes-instrumented/debug"

tasks.withType(Test) {
    jacoco.includeNoLocationClasses = true
}

def coverageSourceDirs = [
        'src/main/java'
]

task jacocoTestReport(type: JacocoReport, dependsOn: "test") {
    group = "Reporting"

    description = "Generate Jacoco coverage reports"

    classDirectories = fileTree(
            dir: 'build/intermediates/classes/debug',
            excludes: ['**/R.class',
                       '**/R$*.class',
                       '**/BuildConfig.*',
                       '**/MainActivity.*']
    )

    sourceDirectories = files(coverageSourceDirs)
    executionData = files('build/jacoco/testDebugUnitTest.exec')
}

jacocoTestReport {
    reports {
        xml.enabled  true
        html.enabled  true
        html.destination file("build/test-results/jacocoHtml")
    }
}

/* This task is used to create offline instrumentation of classes for on-the-fly instrumentation coverage tool like Jacoco. See jacoco classId
     * and Offline Instrumentation from the jacoco site for more info.
     *
     * In this case, some classes mocked using PowerMock were reported as 0% coverage on jacoco & Sonarqube. The issue between PowerMock and jacoco
     * is well documented, and a possible solution is offline Instrumentation (not so well documented for gradle).
     *
     * In a nutshell, this task:
     *  - Pre-instruments the original *.class files
     *  - Puts the instrumented classes path at the beginning of the task's classpath (for report purposes)
     *  - Runs test & generates a new exec file based on the pre-instrumented classes -- as opposed to on-the-fly instrumented class files generated by jacoco.
     *
     * It is currently not implemented to run prior to any other existing tasks (like test, jacocoTestReport, etc...), therefore, it should be called
     * explicitly if Offline Instrumentation report is needed.
     *
     *  Usage: gradle clean & gradle createOfflineInstrTestCoverageReport & gradle jacocoTestReport
     *   - gradle clean //To prevent influence from any previous task execution
     *   - gradle createOfflineInstrTestCoverageReport //To generate *.exec file from offline instrumented class
     *   - gradle jacocoTestReport //To generate html report from newly created *.exec task
     */
task createOfflineTestCoverageReport(dependsOn: ['instrument', 'testDebugUnitTest']) {
    doLast {
        ant.taskdef(name: 'report',
                classname: 'org.jacoco.ant.ReportTask',
                classpath: configurations.jacocoAnt.asPath)
        ant.report() {
            executiondata {
                ant.file(file: "$buildDir.path/jacoco/testDebugUnitTest.exec")
            }
            structure(name: 'Example') {
                classfiles {
                    fileset(dir: "$project.buildDir/intermediates/classes/debug")
                }
                sourcefiles {
                    fileset(dir: 'src/main/java')
                }
            }
            //Uncomment if we want the task to generate jacoco html reports. However, the current script does not exclude files.
            //An alternative is to used jacocoTestReport after this task finishes
            //html(destdir: "$buildDir.path/reports/jacocoHtml")
        }
    }
}

/*
 * Part of the Offline Instrumentation process is to add the jacoco runtime to the class path along with the path of the instrumented files.
 */
gradle.taskGraph.whenReady { graph ->
    if (graph.hasTask(instrument)) {
        tasks.withType(Test) {
            doFirst {
                systemProperty 'jacoco-agent.destfile', buildDir.path + '/jacoco/testDebugUnitTest.exec'
                classpath = files(offline_instrumented_outputDir) + classpath + configurations.jacocoRuntime
            }
        }
    }
}

/*
 *  Instruments the classes per se
 */
task instrument(dependsOn:'compileDebugUnitTestSources') {
    doLast {
        println 'Instrumenting classes'

        ant.taskdef(name: 'instrument',
                classname: 'org.jacoco.ant.InstrumentTask',
                classpath: configurations.jacocoAnt.asPath)

        ant.instrument(destdir: offline_instrumented_outputDir) {
            fileset(dir: "$buildDir.path/intermediates/classes/debug")
        }
    }
}

    Usage

    • The script can be copied into a separate file. For instance: jacoco.gradle

    • Reference the jacoco file in your build.gradle. For instance: apply from: jacoco.gradle

    • Ensure proper dependencies: jacocoAnt 'org.jacoco:org.jacoco.ant:0.8.1:nodeps'

    • In command line run: gradle clean & gradle createOfflineTestCoverageReport & gradle jacocoTestReport

      gradle clean will wipe out any previous gradle execution artifacts

      gradle createOfflineTestCoverageReport will create offline instrumentation, change order of classpath, generate .exec file

      gradle jacocoTestReport will run test and generate jacoco report based on previously generated .exec file

    Feeling Lost?

    I've put together a github Jacoco Powermock Android project with sample scripts to reproduce and fix the issue. It also contains more information about the solution.

    REFERENCE

    https://github.com/powermock/powermock/wiki/Code-coverage-with-JaCoCo
https://www.jacoco.org/jacoco/trunk/doc/classids.html
https://www.jacoco.org/jacoco/trunk/doc/offline.html
https://github.com/powermock/powermock-examples-maven/tree/master/jacoco-offline
https://automated-testing.info/t/jacoco-offline-instrumentations-for-android-gradle/20121
https://stackoverflow.com/questions/41370815/jacoco-offline-instrumentation-gradle-script/42238982#42238982
https://groups.google.com/forum/#!msg/jacoco/5IqM4AibmT8/-x5w4kU9BAAJ