I've run into a situation where install4j v4.2.8 runs fine and generates installers when using the GUI and when invoking install4jc from the command line, but fails with the following message when invoked via the install4j ant task:
install4j: compilation failed. Reason: java.lang.OutOfMemoryError: PermGen space
Using visualvm, I determined that the ant task is forking a separate JVM, which appears to fail when the PermGen usage hits about 88MB. The install4j ant task docs don't say anything about being able to pass VM args (such as -XX:MaxPermSize=256MB) through to the forked JVM, but I'm wondering if that or something similar would be an option.
You can adjust the VM parameters for the command line compiler in the file [install4j installation directory]/bin/install4j.vmoptions. Just increase the -Xmx value there and add the -XX:MaxPermSize VM parameter.
Prior to install4j 5.0.7, there was a bug related to includes of relative files in .vmoptions files. For these older versions, remove the include to install4j.vmoptions in install4jc.vmoptions and add the VM parameters directly there.