Unable to deploy large Worklight app on Tomcat using wladm Ant task

I am using Worklight version 6.2.0.01.20141002-2218 (studio, server, ant tasks) and Tomcat 7.0.55 to host the Worklight server (everything is running on Windows 8.1). As part of our CI build we build the latest application (.wlapp) and adapters (.adapter) with Ant (version 1.8.4) that we then deploy to a Tomcat instance. The problem we see is that for .wlapp files that are larger than approximately 2 MBs we get the following error (IP and port hidden):

[wladm] Oct 06, 2014 1:42:55 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
[wladm] INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error
[wladm] Oct 06, 2014 1:42:55 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
[wladm] INFO: Retrying request
[wladm] Oct 06, 2014 1:42:55 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
[wladm] INFO: I/O exception (java.net.SocketException) caught when processing request: Software caused connection abort: socket write error
[wladm] Oct 06, 2014 1:42:55 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
[wladm] INFO: Retrying request
[wladm] Oct 06, 2014 1:42:55 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
[wladm] INFO: I/O exception (java.net.SocketException) caught when processing request: Software caused connection abort: socket write error
[wladm] Oct 06, 2014 1:42:55 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
[wladm] INFO: Retrying request
[wladm] Error accessing http://XXX.XX.XX.XXX:XXXX/wladmin/management-apis/1.0/runtimes/MyApp/applications?locale=en_GB:

BUILD FAILED
D:\XXX\build.xml:200: com.ibm.worklight.admin.restclient.RESTException: Error accessing http://XXX.XX.XX.XXX:XXXX/wladmin/management-apis/1.0/runtimes/MyApp/applications?locale=en_GB: Software caused connection abort: socket write error
    at com.ibm.worklight.admin.restclient.RESTClient.getResponse(RESTClient.java:1062)
    at com.ibm.worklight.admin.restclient.RESTClient.getPOSTResponse(RESTClient.java:1207)
    at com.ibm.worklight.admin.restclient.RESTClient.getPOSTFileResponse(RESTClient.java:1227)
    at com.ibm.worklight.admin.commands.DeployApp.getResponse(DeployApp.java:41)
    at com.ibm.worklight.admin.restclient.ActionClient.execute(ActionClient.java:85)
    at com.ibm.worklight.admin.ant.types.AbstractActionElement.executeCommand(AbstractActionElement.java:76)
    at com.ibm.worklight.admin.ant.types.ActionElement.executeCommands(ActionElement.java:43)
    at com.ibm.worklight.admin.ant.WladmTask.executeCommands(WladmTask.java:441)
    at com.ibm.worklight.admin.ant.WladmTask.execute(WladmTask.java:290)
    at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
    at org.apache.tools.ant.Task.perform(Task.java:348)
    at org.apache.tools.ant.Target.execute(Target.java:392)
    at org.apache.tools.ant.Target.performTasks(Target.java:413)
    at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
    at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
    at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
    at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
    at org.apache.tools.ant.Main.runBuild(Main.java:811)
    at org.apache.tools.ant.Main.startAnt(Main.java:217)
    at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
    at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
Caused by: java.net.SocketException: Software caused connection abort: socket write error
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
    at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:153)
    at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:114)
    at org.apache.http.entity.mime.content.FileBody.writeTo(FileBody.java:105)
    at org.apache.http.entity.mime.HttpMultipart.doWriteTo(HttpMultipart.java:206)
    at org.apache.http.entity.mime.HttpMultipart.writeTo(HttpMultipart.java:224)
    at org.apache.http.entity.mime.MultipartEntity.writeTo(MultipartEntity.java:183)
    at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:96)
    at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
    at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:120)
    at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:263)
    at org.apache.http.impl.conn.AbstractClientConnAdapter.sendRequestEntity(AbstractClientConnAdapter.java:227)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:255)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
    at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:645)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:464)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
    at com.ibm.worklight.admin.restclient.RESTClient.getResponse(RESTClient.java:951)
    ... 25 more

Note that in the server logs there is no trace of any error. The part of the Ant build file that relates to this is:

<target name="setup-worklight-tasks">
    <taskdef resource="com/worklight/ant/defaults.properties">
        <classpath>
            <pathelement location="${worklight_ant_builder_path}"/>
        </classpath>
    </taskdef>
    <taskdef resource="com/worklight/ant/deployers/antlib.xml">
        <classpath>
            <pathelement location="${worklight_ant_deployer_path}"/>
        </classpath>
    </taskdef>
</target>

<target name="deploy" depends="setup-worklight-tasks">
    <wladm url="${admin_services_url}" user="${admin_services_user}" password="${admin_services_pass}" secure="false">
        <deploy-app runtime="${runtime_name}" file="bin/MyApp-all.wlapp"/>
    </wladm>
</target>

Testing with smaller .wlapp files does not pose any problems resulting in successfully deployed apps. Considering that Tomcat has a default maximum POST size limit of 2 MBs I initially thought that the problem was with my connectors' configuration. However after adapting all connectors for anything that could relate to this, I continue to get the error. Here is the configuration for my connectors in Tomcat's server.xml:

...
<Connector port="9080" protocol="HTTP/1.1"
           connectionTimeout="20000"
           redirectPort="9443" maxPostSize="-1" maxSavePostSize="-1" maxHttpHeaderSize="2097152" socketBuffer="-1" bufferSize="20480"/>

<Connector port="9443" protocol="HTTP/1.1" SSLEnabled="true"
           maxThreads="150" scheme="https" secure="true"
           clientAuth="false" sslProtocol="TLS"
           keystoreFile="D:/IBM/apache-tomcat-7.0.55/security/server2_withchain.p12"
           keystoreType="PKCS12"
           keyPass="myPass" maxPostSize="-1" maxSavePostSize="-1" maxHttpHeaderSize="2097152" socketBuffer="-1" bufferSize="20480"/>               

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"  maxPostSize="-1" maxSavePostSize="-1" maxHttpHeaderSize="2097152" socketBuffer="-1" bufferSize="20480"/>
...

An additional thing I tested was to do the deployment through the Worklight console UI which works fine for any size of .wlapp files. An interesting point I found related to using the Worklight console is (from checking Tomcat's access logs) that the relevant URL is:

XXX.XX.XX.XXX - demo [06/Oct/2014:13:48:05 +0200] "POST /wladmin/management-apis/1.0/runtimes/MyApp/applications?async=true HTTP/1.1" 200 602
0:0:0:0:0:0:0:1 - demo [06/Oct/2014:13:48:05 +0200] "POST /worklightconsole/services/management-apis/1.0/runtimes/MyApp/applications?async=true HTTP/1.1" 200 602

The interesting thing being the presence of the async=true parameter. Taking this into account and considering that normally a "Connection reset by peer: socket write error" normally indicates a problem on the client's side, I'm thinking that this could be an issue with the wladm task that unexpectedly closes the connection (a step that gets avoided when deploying through the UI due to the async=true parameter).

Has anyone faced a similar problem with using the wladm task when deploying to Tomcat?

Update 1:

I did myself the POST of the .wlapp file (using Chrome's POSTman) and the upload plus deployment worked fine. This definitely points to a problem in the wladm ant task.

Update 2:

I replaced in my Ant CI build process the use of wladm with a call to cURL to do the app and adapter files' POST to the Worklight server. I can confirm that using cURL works like a charm, meaning that indeed the problem is with wladm (more likely the bundled Apache HTTP client it uses). Here is the updated part of my Ant build file (I define a boolean flag to use cURL or wladm):

...
<target name="deploy"  unless="deploy_done" depends="build, setup-worklight-tasks" description="Deploy adapters and apps">
    <if>
        <available file="adapters" type="dir"/>
        <then>
            <echo message="Deploying adapters" level="info"/>
            <foreach target="deploy-adapter" param="adapterPath" inheritall="true">
                <path>
                    <dirset dir="adapters" casesensitive="yes">
                        <include name="*"/>
                    </dirset>
                </path>
            </foreach>
        </then>
    </if>
    <echo message="Deploying applications" level="info"/>
    <foreach target="deploy-app" param="appPath" inheritall="true">
        <path>
            <dirset dir="apps" casesensitive="yes">
                <include name="*"/>
            </dirset>
        </path>
    </foreach>
    <property name="deploy_done" value="true"/> 
</target>

<target name="post-curl">
    <exec executable="curl" failonerror="true" outputproperty="curl-output" errorproperty="curl-output-error">
        <arg value="-u"/>
        <arg value="${admin_services_user}:${admin_services_pass}"/>
        <arg value="-F"/>
        <arg value="FILE=@${post-curl.file}"/>
        <arg value="${admin_services_url}/management-apis/1.0/runtimes/${runtime_name}/${post-curl.urlPart}"/>
    </exec>
    <if>
        <not><contains string="${curl-output}" substring="SUCCESS"/></not>
        <then>
            <echo level="error">${curl-output}</echo>
            <echo level="error">${curl-output-error}</echo>
            <fail message="cURL upload failed"/>
        </then>
    </if>
</target>

<target name="deploy-app">
    <basename property="appName" file="${appPath}"/>
    <if>
        <equals arg1="${admin_deploy_curl}" arg2="true"/>
        <then>
            <echo message="Deploying app [${appName}] using cURL..." level="info"/>
            <antcall target="post-curl" inheritAll="true">
                <param name="post-curl.file" value="bin/${appName}-all.wlapp"/>
                <param name="post-curl.urlPart" value="applications"/>
            </antcall>
        </then>
        <else>
            <echo message="Deploying app [${appName}] using wladm..." level="info"/>
            <wladm url="${admin_services_url}" user="${admin_services_user}" password="${admin_services_pass}" secure="false">
                <deploy-app runtime="${runtime_name}" file="bin/${appName}-all.wlapp"/>
            </wladm>
        </else>
    </if>       
    <echo message="App [${appName}] deployed." level="info"/>
</target>

<target name="deploy-adapter">
    <basename property="adapterName" file="${adapterPath}"/>
    <if>
        <equals arg1="${admin_deploy_curl}" arg2="true"/>
        <then>
            <echo message="Deploying adapter [${adapterName}] using cURL..." level="info"/>
            <antcall target="post-curl" inheritAll="true">
                <param name="post-curl.file" value="bin/${adapterName}.adapter"/>
                <param name="post-curl.urlPart" value="adapters"/>
            </antcall>
        </then>
        <else>
            <echo message="Deploying adapter [${adapterName}] using wladm..." level="info"/>                
            <wladm url="${admin_services_url}" user="${admin_services_user}" password="${admin_services_pass}" secure="false">
                <deploy-adapter runtime="${runtime_name}" file="bin/${adapterName}.adapter"/>
            </wladm>
        </else>
    </if>       
    <echo message="Adapter [${adapterName}] deployed." level="info"/>
</target>
...

Solution

The problem is introduced by a new behaviour in Tomcat 7.0.55. Citing its change log: "Add a new limit, defaulting to 2MB, for the amount of data Tomcat will swallow for an aborted upload. The limit is configurable by maxSwallowSize attribute of an HTTP connector."

So, there are three possible workarounds:

Change all <Connector> declarations, in Tomcat's conf/server.xml, by adding an attribute maxSwallowSize="100000000" (or whatever upper limit on the apps and adapters size you can impose).
Or use the Worklight Console UI to deploy the application or adapter.
Or use the curl program to deploy the application or adapter.