HTTP Vs Vanilla sockets for uploading large binary file (50-200 MB) from Android device over a single hop Wi-Fi Network

Question

Is there a substantial overhead of using HTTP over plain sockets (Java on Android) to send a large (50-200 MB) file [file is on the SD card] from an Android device to a Linux server over a Wi-Fi network.

In my current prototype I'm using CherryPy-3.2.0 to implement my HTTP server. I'm running Android 2.3.3 on a Nexus one as my client.

Currently it's taking around ~100 seconds** (on slower network 18 Mbps*) and ~50 seconds (on a faster 54 Mbps*) Wi-Fi network to upload a 50 MB binary file.

NOTE:
*I'm using WifiInfo.getLinkSpeed() to measure the network link speed

** This is the time difference before and after the HTTPClient.execute(postRequest)

Any other ideas regarding other expensive operations that may have a substantial part in the total time apart from the network and how to reduce this time would be appreciated.

Thanks.

EDIT - HTTP post code on Android

private void doHttpPost(String fileName) throws Exception{

    HttpParams httpParameters = new BasicHttpParams();

    // Set the timeout in milliseconds until a connection is established.
    int timeoutConnection = 9000000;
    HttpConnectionParams.setConnectionTimeout(httpParameters, timeoutConnection);
    // Set the default socket timeout (SO_TIMEOUT) 
    // in milliseconds which is the timeout for waiting for data.
    int timeoutSocket = 9000000;
    HttpConnectionParams.setSoTimeout(httpParameters, timeoutSocket);

    HttpClient client = new DefaultHttpClient(httpParameters);

    client.getParams().setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.RFC_2109);

    HttpPost postRequest = new HttpPost();
    postRequest.setURI(new URI("http://192.168.1.107:9999/upload/"));

    MultipartEntity multiPartEntity = new MultipartEntity();
    multiPartEntity.addPart("myFile", new FileBody(new File(fileName)));
    postRequest.setEntity(multiPartEntity);

    long before = TrafficStats.getTotalTxBytes();
    long start = System.currentTimeMillis();
    HttpResponse response = client.execute(postRequest);
    long end = System.currentTimeMillis(); 
    long after = TrafficStats.getTotalTxBytes();

    Log.d(LOG_TAG, "HTTP Post Execution took " + (end - start) + " ms.");


    if( before != TrafficStats.UNSUPPORTED && after != TrafficStats.UNSUPPORTED)
        Log.d(LOG_TAG, (after-before) + " bytes transmitted to the server");
    else
        Log.d(LOG_TAG, "This device doesnot support Network Traffic Stats");

    HttpEntity responseEntity = response.getEntity();


    if (responseEntity != null) {
        responseEntity.consumeContent();
        Log.d(LOG_TAG, "HTTP Post Response " + response.getEntity().getContent().toString() );
    }

    client.getConnectionManager().shutdown(); 

}

EDIT 2: Based on the results reported by this tool it looks like the SD card read speed is not an issue. So it may either be the HttpClient library or something else.

Answer 1

Overhead on HTTP connection comes from the headers that it sends along with your data (which is basically a constant). So the more data you send, the less the headers 'hurt you'. However, the much more important aspect to consider is encoding.

For example, if you are sending non-ASCII data, paired with a mime type of application/x-www-form-urlencoded you run the risk of exploding the input size because non-ASCII characters must be escaped.

From the spec:

The content type "application/x-www-form-urlencoded" is inefficient for sending large quantities of binary data or text containing non-ASCII characters. The content type "multipart/form-data" should be used for submitting forms that contain files, non-ASCII data, and binary data.

The alternative is multipart/form-data which efficient for binary data. So, make sure your application is using this MIME type (you can even probably check this on your server logs).

Another method which can considerably reduce your upload time is compression. If you are uploading data which isn't already compressed (most image and video formats are already compressed) try adding gzip compression to your uploads. Another post shows the details of setting this up in android.

If your data is of a specific format (say an image), you can look into lossless compression algorithms for your type of data (png for images, FLAC for audio, etc.). Compression always comes at the price of CPU (battery), so keep that in mind.

Remember:

Don't optimize something until you know its the bottleneck. Maybe your server's connection is slow, maybe you can't read from the android file system fast enough to push your data to the network. Run some tests and see what works.

If it were me, I would not implement the straight tcp approach. Just my 2 cents, good luck!