How to obfuscate with ProGuard but keep names readable while testing?

Question

I'm in a pre-release stage with my app where I started compiling release builds assembleRelease instead of assembleDebug. However the obfuscation breaks things and it's hard to decipher what's what. Debugging is almost impossible, even with line numbers kept the variables classes are unreadable. While the release build is not stable I'd like to make obfuscation less of a pain, but it should still behave as it was fully obfuscated.

Usually a ProGuarded release converts names from

net.twisterrob.app.pack.MyClass

to

b.a.c.b.a

with which reflection and Android layout/menu resources can break, if they encountered classes that we didn't keep the names of.

It would be really helpful for pre-release testing to be able to obfuscate the code, but "not that much", like converting names from

net.twisterrob.app.pack.MyClass

to

net.twisterrob.app.pack.myclass // or n.t.a.p.MC or anything in between :)

The proguard -dontobfuscate of course helps, but then it makes all broken stuff work again because class names are correct.

What I'm looking for will break what would be broken with full obfuscation, but at the same time it's easy to figure out what's what without using the mapping.txt because the names are kept human readable.

I was looking around http://proguard.sourceforge.net/manual/usage.html#obfuscationoptions but the -*dictionary options don't seem to be doing this.

I would be fine to generate a renaming file myself (it would be just running through all the classes and give them a toLowerCase or something):

net.twisterrob.app.pack.MyClassA -> myclassa
net.twisterrob.app.pack.MyClassB -> myclassb

The question is then how would I feed such a file to ProGuard and what is the format?

Answer 1

So it looks like I've managed to skip over the option -applymapping in the very section I linked.

TL;DR

Jump to Implementation / details section and copy those two blocks of Gradle/Groovy code to your Android subproject's build.gradle file.

mapping.txt

The format of mapping.txt is pretty simple:

full.pack.age.Class -> obf.usc.ate.d:
    Type mField -> mObfusc
    #:#:Type method(Arg,s) -> methObfusc
kept.Class -> kept.Class:
    Type mKept -> mKept
    #:#:Type kept() -> kept

The shrinked classes and members are not listed at all. So all information available, if I can generate the same or transform this, there's a pretty good chance of success.

Solution 1: Dump all classes [failed]

I've tried to generate an input mapping.txt based the current classpath that is passed to proguard (-injars). I loaded all the classes in an URLClassLoader which had all the program jars as well as the libraryjars (to resolve super classes for example). Then iterated through each class and each declared member and output a name I would have liked to use.

There was a big problem with this: this solution contained an obfuscated name for each and every renameable thing in the app. The problem here is that the -applymapping takes things literally and tries to apply all mappings in the input mapping file, ignoring the -keep rules, leading to warnings about conflicted renames. So I gave up on this path, because I didn't want to duplicate the proguard config, nor wanted to implement the proguard config parser myself.

Solution 2: Run proguardRelease twice [failed]

Based on the above fail, I thought of another solution which would make use of all the configuration and keeps there are. The flow is the following:

• let proguardRelease do it's job
  this outputs the source mapping.txt
• transform the mapping.txt into a new file
• duplicate proguardRelease gradle task and run it with the transformed mapping

The problem with this that it's really complicated to duplicate the whole task, with all it's inputs, outputs, doLast, doFirst, @TaskAction, etc... I actually started on this route anyway, but it soon joined into the 3rd solution.

Solution 3: Use proguardRelease's output [success]

While trying to duplicate the whole task and analyzing proguard/android plugin code I realized it would be much easier to just simulate what proguardRelease is doing again. Here's the final flow:

• let proguardRelease do it's job
  this outputs the source mapping.txt
• transform the mapping.txt into a new file
• run proguard again with the same config,
  but this time using my mapping file for renames

The result is what I wanted:
(example the pattern is <package>.__<class>__.__<field>__ with class and field names with inverted case)

java.lang.NullPointerException: Cannot find actionView! Is it declared in XML and kept in proguard?
        at net.twisterrob.android.utils.tools.__aNDROIDtOOLS__.__PREPAREsEARCH__(AndroidTools.java:533)
        at net.twisterrob.inventory.android.activity.MainActivity.onCreateOptionsMenu(MainActivity.java:181)
        at android.app.Activity.onCreatePanelMenu(Activity.java:2625)
        at android.support.v4.app.__fRAGMENTaCTIVITY__.onCreatePanelMenu(FragmentActivity.java:277)
        at android.support.v7.internal.view.__wINDOWcALLBACKwRAPPER__.onCreatePanelMenu(WindowCallbackWrapper.java:84)
        at android.support.v7.app.__aPPcOMPATdELEGATEiMPLbASE$aPPcOMPATwINDOWcALLBACK__.onCreatePanelMenu(AppCompatDelegateImplBase.java:251)
        at android.support.v7.app.__aPPcOMPATdELEGATEiMPLv7__.__PREPAREpANEL__(AppCompatDelegateImplV7.java:1089)
        at android.support.v7.app.__aPPcOMPATdELEGATEiMPLv7__.__DOiNVALIDATEpANELmENU__(AppCompatDelegateImplV7.java:1374)
        at android.support.v7.app.__aPPcOMPATdELEGATEiMPLv7__.__ACCESS$100__(AppCompatDelegateImplV7.java:89)
        at android.support.v7.app.__aPPcOMPATdELEGATEiMPLv7$1__.run(AppCompatDelegateImplV7.java:123)
        at android.os.Handler.handleCallback(Handler.java:733)

Or notice the underscores here:

Implementation / details

I tried to make it as simple as possible, while keeping maximum flexibility. I call it unfuscation, becuase it's undoing proper obfuscation, but still considered obfuscation in terms of reflection for example.

I implemented a few guards because the 2nd round makes a few assumptions. Obivously if there's no obfuscation, there's no need to unfuscated. Also it's almost pointless to unfuscate (and may be accidentally released) if debuging is turned off since unfuscation helps most inside the IDE. If the the app is tested and obfuscated, the interals of AndroidProguardTask is using the mapping file and I didn't want to deal with that now.

So I went ahead and created an unfuscate task, which does the transformation and runs proguard. Sadly the proguard configuration is not exposed in proguard.gradle.ProguardTask, but when did that stop anyone?! :)

There's one drawback, it takes double time to proguard it, which I guess is worth it if you really need to debug it.

Here's the android hooking code for Gradle:

afterEvaluate {
    project.android.applicationVariants.all { com.android.build.gradle.api.ApplicationVariant variant ->
        Task obfuscateTask = variant.obfuscation
        def skipReason = [ ];
        if (obfuscateTask == null) { skipReason += "not obfuscated" }
        if (!variant.buildType.debuggable) { skipReason += "not debuggable" }
        if (variant.testVariant != null) { skipReason += "tested" }
        if (!skipReason.isEmpty()) {
            logger.info("Skipping unfuscation of {} because it is {}", variant.name, skipReason);
            return;
        }

        File mapping = variant.mappingFile
        File newMapping = new File(mapping.parentFile, "unmapping.txt")

        Task unfuscateTask = project.task("${obfuscateTask.name}Unfuscate") {
            inputs.file mapping
            outputs.file newMapping
            outputs.upToDateWhen { mapping.lastModified() <= newMapping.lastModified() }
            doLast {
                java.lang.reflect.Field configField =
                        proguard.gradle.ProGuardTask.class.getDeclaredField("configuration")
                configField.accessible = true
                proguard.Configuration config = configField.get(obfuscateTask) as proguard.Configuration
                if (!config.obfuscate) return; // nothing to unfuscate when -dontobfuscate

                java.nio.file.Files.copy(mapping.toPath(), new File(mapping.parentFile, "mapping.txt.bck").toPath(),
                        java.nio.file.StandardCopyOption.REPLACE_EXISTING)
                logger.info("Writing new mapping file: {}", newMapping)
                new Mapping(mapping).remap(newMapping)

                logger.info("Re-executing {} with new mapping...", obfuscateTask.name)
                config.applyMapping = newMapping // use our re-written mapping file
                //config.note = [ '**' ] // -dontnote **, it was noted in the first run

                LoggingManager loggingManager = getLogging();
                // lower level of logging to prevent duplicate output
                loggingManager.captureStandardOutput(LogLevel.WARN);
                loggingManager.captureStandardError(LogLevel.WARN);
                new proguard.ProGuard(config).execute();
            }
        }
        unfuscateTask.dependsOn obfuscateTask
        variant.dex.dependsOn unfuscateTask
    }
}

The other part of the whole is the transformation. I managed to quickly compose an all-matching regex pattern, so it is pretty simple. You can safely ignore the class structure and the remap method. The key is processLine which is called for each line. The line is split to parts, the text before and after the obfuscated name stays as is (two substrings) and the name is changed in the middle. Change to return statement in unfuscate to suit your needs.

class Mapping {
    private static java.util.regex.Pattern MAPPING_PATTERN =
            ~/^(?<member>    )?(?<location>\d+:\d+:)?(?:(?<type>.*?) )?(?<name>.*?)(?:\((?<args>.*?)\))?(?: -> )(?<obfuscated>.*?)(?<class>:?)$/;
    private static int MAPPING_PATTERN_OBFUSCATED_INDEX = 6;

    private final File source
    public Mapping(File source) {
        this.source = source
    }

    public void remap(File target) {
        target.withWriter { source.eachLine Mapping.&processLine.curry(it) }
    }

    private static void processLine(Writer out, String line, int num) {
        java.util.regex.Matcher m = MAPPING_PATTERN.matcher(line)
        if (!m.find()) {
            throw new IllegalArgumentException("Line #${num} is not recognized: ${line}")
        }
        try {
            def originalName = m.group("name")
            def obfuscatedName = m.group("obfuscated")
            def newName = originalName.equals(obfuscatedName) ? obfuscatedName : unfuscate(originalName, obfuscatedName)
            out.write(line.substring(0, m.start(MAPPING_PATTERN_OBFUSCATED_INDEX)))
            out.write(newName)
            out.write(line.substring(m.end(MAPPING_PATTERN_OBFUSCATED_INDEX)))
            out.write('\n')
        } catch (Exception ex) {
            StringBuilder sb = new StringBuilder("Line #${num} failed: ${line}\n");
            0.upto(m.groupCount()) { sb.append("Group #${it}: '${m.group(it)}'\n") }
            throw new IllegalArgumentException(sb.toString(), ex)
        }
    }

    private static String unfuscate(String name, String obfuscated) {
        int lastDot = name.lastIndexOf('.') + 1;
        String pkgWithDot = 0 < lastDot ? name.substring(0, lastDot) : "";
        name = 0 < lastDot ? name.substring(lastDot) : name;
        // reassemble the names with something readable, but still breaking changes
        // pkgWithDot will be empty for fields and methods
        return pkgWithDot + '_' + name;
    }
}

Possible unfuscations

You should be able to apply a transformation to package names, but I didn't test that.

// android.support.v4.a.a, that is the original obfuscated one
return obfuscated;

// android.support.v4.app._Fragment
return pkgWithDot + '_' + name;

// android.support.v4.app.Fragment_a17d4670
return pkgWithDot + name + '_' + Integer.toHexString(name.hashCode());

// android.support.v4.app.Fragment_a
return pkgWithDot + name + '_' + afterLastDot(obfuscated)

// android.support.v4.app.fRAGMENT
return pkgWithDot + org.apache.commons.lang.StringUtils.swapCase(name);
// needs the following in build.gradle:
buildscript {
    repositories { jcenter() }
    dependencies { classpath 'commons-lang:commons-lang:2.6' }
}

// android.support.v4.app.fragment
return pkgWithDot + name.toLowerCase();

WARNING: irreversible transformations are error-prone. Consider the following:

class X {
    private static final Factory FACTORY = ...;
    ...
    public interface Factory {
    }
}
// notice how both `X.Factory` and `X.FACTORY` become `X.factory` which is not allowed.

Of course all of the above transformations can be tricked in one way or another, but it's less likely with uncommon pre-postfixes and text-transformations.