Downloading large file in ColdFusion using CFHTTP

Question

I'm trying to download a large (600MB) binary file to the server using CFHTTP on Coldfusion 8:

<cfhttp 
  method="get" 
  url="#fileURL#" 
  path="#filePath#" 
  file="#fileName#" 
  timeout="600" 
  getasbinary="auto"
>

It's working fine for smaller files (100something MB) but for the large one's I'm getting the Server 500 error: "java.lang.OutOfMemoryError: Java heap space".

The file is being downloaded from a Dropbox folder - so only available option is to use HTTP GET.

Does anyone have idea how to download it, so it wouldn't kill the server or timeout?

Answer 1

You can do this by calling Java from CF code. The buffered input and output stream classees are designed to hold onto chunks of data, rather than the whole thing, avoiding OutOfMemory errors.

getByteArray() is a helper method because there's not way to declare something like byte buf[]=new byte[1024]; in CF directly.

In the example change the source and destination variables.

Example

<cfset source = 'protocol://domain/path/to/file.ext'>
<cfset destination = getDirectoryFromPath(getCurrentTemplatePath()) & listlast(source, "/")>
<cffunction name="getByteArray" access="private" returnType="binary" output="no">
    <cfargument name="size" type="numeric" required="true"/>
    <cfset var emptyByteArray =
        createObject("java", "java.io.ByteArrayOutputStream").init().toByteArray()/>
    <cfset var byteClass = emptyByteArray.getClass().getComponentType()/>
    <cfset var byteArray =
        createObject("java","java.lang.reflect.Array").newInstance(byteClass, arguments.size)/>
    <cfreturn byteArray/>
</cffunction>

<cfscript>
uri = createObject("java", "java.net.URL").init(source);
uis = uri.openStream();
bis = createObject("java", "java.io.BufferedInputStream").init(uis);
fos = createObject("java", "java.io.FileOutputStream").init(destination);
bos = createObject("java", "java.io.BufferedOutputStream").init(fos);
buffer = getByteArray(1024);
len = bis.read(buffer);
while(len > 0) {
    bos.write(buffer,0,len);
    len = bis.read(buffer);
}
bos.close();
bis.close();
fos.close();
uis.close();
</cfscript>