OutOfMemory on big HTTP MultipartRequestEntity via apache HttpClient 3.1

Question

I'm actually trying to upload a file (>5M) using apache HttpClient 3.1 via MultipartRequestEntity with java 1.4 in a low memory environment, so increasing java heap is not possible

I have a file splitted in many parts, I just need to append all the data of each part into the request, I could get the file without splitting too

The real problem is that the request I'm building, is producing the memory to overflow, is there any way to avoid this and be able to send this request at once?

This is what I actually have, I would need to append the bytes to the part (a kind of part[0].write(data)), or get a stream to write it without loading into the memory, I don't know if it is actually possible

    HttpClient client = new HttpClient();
    PostMethod post = new PostMethod(url);

    post.setContentChunked(true);

    try {
        Part[] parts = new Part[fileNames.size()];

        String fileName = "sample.pdf";

        for(int i = 0; i<fileNames.size();i++){
            File file = Util.getFile(fileNames.get(i));
            FileInputStream is = new FileInputStream(file);
            byte[] fileBytes = IOUtils.toByteArray(is);
            parts[i] = new FilePart("file", new ByteArrayPartSource(fileName, fileBytes) , "application/octet-stream", post.getRequestCharSet()); 
            is.close();
            is = null;
            fileBytes = null;
        }

     MultipartRequestEntity requestEntity = new MultipartRequestEntity(parts, post.getParams());

     post.setRequestEntity(requestEntity);


     int statusCode = client.executeMethod(post);

Another possibility is with HttpUrlConnection writing a BufferedWriter, but this will also give OutOfMemory when writing on it

Answer 1

I could have done this with File java file class directly, but the problem was that I have not a java File class, instead I had to implement PartSource to read my special file class and set it to read after joining all of my file parts in one single file obligatorily. Usually you should do this:

parts[0] = new FilePart("file", new FilePartSource(file) , "application/octet-stream", postMethod.getRequestCharSet()); 

You need a File if you don't want to get the whole data buffered in memory for the request

Now I have this solved