I want to repackage apache's httpclient lib to ship it with an android app (like https://code.google.com/p/httpclientandroidlib/ but with HttpClient 4.3.1)
Therefore, I downloaded the httpclient 4.3.1 jar (includes all its dependencies) by hand and used jarjar to repackage it:
x@x$: cd libs && for f in *.jar; do java -jar ../jarjar-1.4.jar process ../rules.txt $f out/my-$f; done
with rules.txt:
rule org.apache.http.** my.repackaged.org.apache.http.@1
Then I used ant to put the output together:
<project name="MyProject" default="merge" basedir=".">
<target name="merge">
<zip destfile="my-org-apache-httpclient-4.3.1.jar">
<zipgroupfileset dir="libs/out" includes="*.jar"/>
</zip>
</target>
</project>
I can use that file to develop and test my app, but if I deploy it on android, it throws an exception s/th like that it cannot find my.repackaged.org.apache.logging.log4j.something referenced by my.package.org.apache.logging.whatEver.
So, now I want to strip out any dependency on commons-logging by using bytecode manipulation. This has been done before: http://sixlegs.com/blog/java/dependency-killer.html
But I wonder how I actually do it? There are only dependencies on org.apache.commons.logging.Log:
x$x$: java -jar jarjar-1.4.jar find jar my-org-apache-httpclient-4.3.1.jar commons-logging-1.1.3.jar
my/http/impl/execchain/ServiceUnavailableRetryExec -> org/apache/commons/logging/Log
my/http/impl/execchain/RetryExec -> org/apache/commons/logging/Log
my/http/impl/execchain/RedirectExec -> org/apache/commons/logging/Log
my/http/impl/execchain/ProtocolExec -> org/apache/commons/logging/Log
...
I think the way to go is, to remove these dependencies and replace it with an own implementation like he did here https://code.google.com/p/httpclientandroidlib/ . Therefore, I made a new maven project with only one class with provided scope for the commons-logging that implements org.apache.commons.logging.Log interface and just delefates to the android.utils.Log:
MyLog implements org.apache.commons.logging.Log {}
in the package my.log and I packaged that in my-log-1.0.0.jar. I put that jar into the same folder as the repackaged httpclient-jars and used ant as mentioned above to package all together in my-org-apache-httpclient-4.3.1.jar.
Approach 1
I tried to use jarjar again:
java -jar jarjar-1.4.jar process rules2.txt my-org-apache-httpclient-4.3.1.jar my-org-apache-httpclient-4.3.1-without-logging-dep.jar
with rules2.txt:
rule my.repackaged.commons.logging.** my.log.@1
but that does not work. The exception that it cannot find my.repackaged.org.apache.logging.log4j.something referenced by my.package.org.apache.logging.whatEver is still thrown.
Approach 2
I also tried to delete the logging stuff from the final jar and/or repackage the my.repackaged.org.apache.log4j and logging to its original packages:
rules2.txt v2:
rule my.repackaged.org.apache.log4j.** org.apache.log4j.@1
rule my.repackaged.org.apache.logging.** org.apache.logging.@1
but that also is still throwing the excpetion: my.repackaged.org.apache.logging.log4j.something referenced by my.package.org.apache.logging.whatEver
QUESTION
How can I kill/replace that commons-logging dependencies and get rid of the Exception?
If a program depends on a library it usually means that it uses methods of the library. Removing a dependency is therefore not a simple task. You effectively want to take away code that is - at least formally - required by the program.
There are three ways of removing dependencies:
None of these ways are really pretty. All of them can require a lot of work. None are guaranteed to work without side effects.
I will describe my solution by presenting the files and steps I used to solve the problem. To reproduce, you will need the following files (in a single directory):
lib/xxx-v.v.v.jar: The library jars (httpclient and dependencies, excluding commons-logging-1.1.3.jar)
jarjar-1.4.jar: Used for repackaging the jars
rules.txt: The jarjar rules
rule org.apache.http.** my.http.@1
rule org.apache.commons.logging.** my.logging.@1
build.xml: Ant build configuration
<project name="MyProject" basedir=".">
<target name="logimpl">
<javac srcdir="java/src" destdir="java/bin" target="1.5" />
<jar jarfile="out/logimpl.jar" basedir="java/bin" />
</target>
<target name="merge">
<zip destfile="httpclient-4.3.1.jar">
<zipgroupfileset dir="out" includes="*.jar"/>
</zip>
</target>
</project>
java/src/Log.java
package my.logging;
public interface Log {
public boolean isDebugEnabled();
public void debug(Object message);
public void debug(Object message, Throwable t);
public boolean isInfoEnabled();
public void info(Object message);
public void info(Object message, Throwable t);
public boolean isWarnEnabled();
public void warn(Object message);
public void warn(Object message, Throwable t);
public boolean isErrorEnabled();
public void error(Object message);
public void error(Object message, Throwable t);
public boolean isFatalEnabled();
public void fatal(Object message);
public void fatal(Object message, Throwable t);
}
java/src/LogFactory.java
package my.logging;
public class LogFactory {
private static Log log;
public static Log getLog(Class<?> clazz) {
return getLog(clazz.getName());
}
public static Log getLog(String name) {
if(log == null) {
log = new Log() {
public boolean isWarnEnabled() { return false; }
public boolean isInfoEnabled() { return false; }
public boolean isFatalEnabled() { return false; }
public boolean isErrorEnabled() {return false; }
public boolean isDebugEnabled() { return false; }
public void warn(Object message, Throwable t) {}
public void warn(Object message) {}
public void info(Object message, Throwable t) {}
public void info(Object message) {}
public void fatal(Object message, Throwable t) {}
public void fatal(Object message) {}
public void error(Object message, Throwable t) {}
public void error(Object message) {}
public void debug(Object message, Throwable t) {}
public void debug(Object message) {}
};
}
return log;
}
}
do_everything.sh
#!/bin/sh
# Repackage library
mkdir -p out
for jf in lib/*.jar; do
java -jar jarjar-1.4.jar process rules.txt $jf `echo $jf | sed 's/lib\//out\//'`
done
# Compile logging implementation
mkdir -p java/bin
ant logimpl
# Merge jar files
ant merge
That's it. Open up a console and execute
cd my_directory && ./do_everything.sh
This will create a folder "out" containing single jar files and "httpclient-4.3.1.jar" which is the final, independent and working jar file. So, what did we just do?
my.http)my.logging instead of org.apache.commons.loggingmy.logging.Log and my.logging.LogFactory).Pretty simple, isn't it? Just read the shell script line by line to discover the single steps. To check whether all dependencies were removed you can run
java -jar jarjar-1.4.jar find class httpclient-4.3.1.jar commons-logging-1.1.3.jar
I tried the generated jar file with SE7 and Android 4.4, it worked in both cases (see below for remarks).
Every class file has a major version and a minor version (both depend on the compiler). The Android SDK requires class files to have a major version less than 0x33 (so everything pre 1.7 / JDK 7). I added the target="1.5" attribute to the ant javac task so the generated class files have a major version of 0x31 and can therefore be included in your Android app.
You're lucky. Logging is (almost always) a one-way operation. It barely causes side effects affecting the main program. That means that removing commons-logging should be possible as it won't affect the functionality of the program.
I chose the second way, bytecode manipulation, which you suggested in your question. The concept is basically just this (A is httpclient, B is commons-logging):
java.lang.Object.java.lang.Object.pop and constant instructions are inserted to repair the VM stack.java.lang.Object.pop and constant instructions are inserted to repair the VM stack.java.lang.Object. This requires the target class (the class containing the accessed field) to be processed.java.lang.Object.As you can see, the idea behind this is to replace all referenced classes with java.lang.Object and to remove all accesses to class members belonging to commons-logging.
I don't know whether this is reliable and I did not test the library after applying the manipulator. But from what I saw (the disassembled class files and no VM errors while loading the class files) I am fairly sure the code works.
I tried to document almost everything the program does. It uses the ASM Tree API which provides rather simple access to the class file structure. And - to avoid unnecessary negative reviews - this is "quick 'n' dirty" code. I did not really test it a lot and I bet there are faster ways of bytecode manipulation. But this program seems to fulfill the OP's needs and that's all I wrote it for.
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Enumeration;
import java.util.List;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
import java.util.jar.JarOutputStream;
import org.objectweb.asm.ClassReader;
import org.objectweb.asm.ClassWriter;
import org.objectweb.asm.Opcodes;
import org.objectweb.asm.Type;
import org.objectweb.asm.tree.AbstractInsnNode;
import org.objectweb.asm.tree.ClassNode;
import org.objectweb.asm.tree.FieldInsnNode;
import org.objectweb.asm.tree.FieldNode;
import org.objectweb.asm.tree.InsnList;
import org.objectweb.asm.tree.InsnNode;
import org.objectweb.asm.tree.MethodInsnNode;
import org.objectweb.asm.tree.MethodNode;
public class DependencyFinder {
public static void main(String[] args) throws IOException {
if(args.length < 2) return;
DependencyFinder df = new DependencyFinder();
df.analyze(new File(args[0]), new File(args[1]), "org.apache.http/.*", "org.apache.commons.logging..*");
}
@SuppressWarnings("unchecked")
public void analyze(File inputFile, File outputFile, String sClassRegex, String dpClassRegex) throws IOException {
JarFile inJar = new JarFile(inputFile);
JarOutputStream outJar = new JarOutputStream(new FileOutputStream(outputFile));
for(Enumeration<JarEntry> entries = inJar.entries(); entries.hasMoreElements();) {
JarEntry inEntry = entries.nextElement();
InputStream inStream = inJar.getInputStream(inEntry);
JarEntry outEntry = new JarEntry(inEntry.getName());
outEntry.setTime(inEntry.getTime());
outJar.putNextEntry(outEntry);
OutputStream outStream = outJar;
// Only process class files, copy all other resources
if(inEntry.getName().endsWith(".class")) {
// Initialize class reader and writer
ClassReader classReader = new ClassReader(inStream);
ClassWriter classWriter = new ClassWriter(0);
String className = classReader.getClassName();
// Check whether to process this class
if(className.matches(sClassRegex)) {
System.out.println("Processing " + className);
// Parse entire class
ClassNode classNode = new ClassNode(Opcodes.ASM4);
classReader.accept(classNode, 0);
// Check super class and interfaces
String superClassName = classNode.superName;
if(superClassName.matches(dpClassRegex)) {
throw new RuntimeException(className + " extends " + superClassName);
}
for(String iface : (List<String>) classNode.interfaces) {
if(iface.matches(dpClassRegex)) {
throw new RuntimeException(className + " implements " + superClassName);
}
}
// Process methods
for(MethodNode method : (List<MethodNode>) classNode.methods) {
Type methodDesc = Type.getMethodType(method.desc);
boolean changed = false;
// Change return type if necessary
Type retType = methodDesc.getReturnType();
if(retType.getClassName().matches(dpClassRegex)) {
retType = Type.getObjectType("java/lang/Object");
changed = true;
}
// Change argument types if necessary
Type[] argTypes = methodDesc.getArgumentTypes();
for(int i = 0; i < argTypes.length; i++) {
if(argTypes[i].getClassName().matches(dpClassRegex)) {
argTypes[i] = Type.getObjectType("java/lang/Object");
changed = true;
}
}
if(changed) {
// Update method descriptor
System.out.print("Changing " + method.name + methodDesc);
methodDesc = Type.getMethodType(retType, argTypes);
method.desc = methodDesc.getDescriptor();
System.out.println(" to " + methodDesc);
}
// Remove method invocations
InsnList insns = method.instructions;
for(int i = 0; i < insns.size(); i++) {
AbstractInsnNode insn = insns.get(i);
// Ignore all other nodes
if(insn instanceof MethodInsnNode) {
MethodInsnNode mnode = (MethodInsnNode) insn;
Type[] cArgTypes = Type.getArgumentTypes(mnode.desc);
Type cRetType = Type.getReturnType(mnode.desc);
if(mnode.owner.matches(dpClassRegex)) {
// The method belongs to one of the classes we want to get rid of
System.out.println("Removing method call " + mnode.owner + "." +
mnode.name + " in " + method.name);
boolean isStatic = (mnode.getOpcode() == Opcodes.INVOKESTATIC);
if(!isStatic) {
// pop instance
insns.insertBefore(insn, new InsnNode(Opcodes.POP));
}
for(int j = 0; j < cArgTypes.length; j++) {
// pop argument on stack
insns.insertBefore(insn, new InsnNode(Opcodes.POP));
}
// Insert a constant value to repair the stack
if(cRetType.getSort() != Type.VOID) {
InsnNode valueInsn = getValueInstruction(cRetType);
insns.insertBefore(insn, valueInsn);
}
// Remove the actual method call
insns.remove(insn);
// Go back one instruction to not skip the next one
i--;
} else {
changed = false;
if(cRetType.getClassName().matches(dpClassRegex)) {
// Change return type
cRetType = Type.getObjectType("java/lang/Object");
changed = true;
}
for(int j = 0; j < cArgTypes.length; j++) {
if(cArgTypes[j].getClassName().matches(dpClassRegex)) {
// Change argument type
cArgTypes[j] = Type.getObjectType("java/lang/Object");
changed = true;
}
}
if(changed) {
// Update method invocation
System.out.println("Patching method call " + mnode.owner + "." +
mnode.name + " in " + method.name);
mnode.desc = Type.getMethodDescriptor(cRetType, cArgTypes);
}
}
} else if(insn instanceof FieldInsnNode) {
// Yeah I lied... we must not ignore all other instructions
FieldInsnNode fnode = (FieldInsnNode) insn;
Type fieldType = Type.getType(fnode.desc);
if(fnode.owner.matches(dpClassRegex)) {
System.out.println("Removing field access to " + fnode.owner + "." +
fnode.name + " in " + method.name);
// Patch code
switch(fnode.getOpcode()) {
case Opcodes.PUTFIELD:
case Opcodes.GETFIELD:
// Pop instance
insns.insertBefore(insn, new InsnNode(Opcodes.POP));
if(fnode.getOpcode() == Opcodes.PUTFIELD) break;
case Opcodes.GETSTATIC:
// Repair stack
insns.insertBefore(insn, getValueInstruction(fieldType));
break;
default:
throw new RuntimeException("Invalid opcode");
}
// Remove instruction
insns.remove(fnode);
i--;
} else {
if(fieldType.getClassName().matches(dpClassRegex)) {
// Change field type
System.out.println("Patching field access to " + fnode.owner +
"." + fnode.name + " in " + method.name);
fieldType = Type.getObjectType("java/lang/Object");
}
// Update field type
fnode.desc = fieldType.getDescriptor();
}
}
}
}
// Process fields
for(FieldNode field : (List<FieldNode>) classNode.fields) {
Type fieldType = Type.getType(field.desc);
if(fieldType.getClassName().matches(dpClassRegex)) {
System.out.print("Changing " + fieldType.getClassName() + " " + field.name);
fieldType = Type.getObjectType("java/lang/Object");
field.desc = fieldType.getDescriptor();
System.out.println(" to " + fieldType.getClassName());
}
}
// Class processed
classNode.accept(classWriter);
} else {
// Nothing changed
classReader.accept(classWriter, 0);
}
// Write class to JAR entry
byte[] bClass = classWriter.toByteArray();
outStream.write(bClass);
} else {
// Copy file
byte[] buffer = new byte[1024 * 64];
int read;
while((read = inStream.read(buffer)) != -1) {
outStream.write(buffer, 0, read);
}
}
outJar.closeEntry();
}
outJar.flush();
outJar.close();
inJar.close();
}
InsnNode getValueInstruction(Type type) {
switch(type.getSort()) {
case Type.INT:
case Type.BOOLEAN:
return new InsnNode(Opcodes.ICONST_0);
case Type.LONG:
return new InsnNode(Opcodes.LCONST_0);
case Type.OBJECT:
case Type.ARRAY:
return new InsnNode(Opcodes.ACONST_NULL);
default:
// I am lazy, I did not implement all types
throw new RuntimeException("Type not implemented: " + type);
}
}
}