Search code examples
javamultithreadingfileasynchronousjava-6

Downloading Multiple Files Parallelly or Asynchronously in Java


Here I am trying to download multiple files one after another:

Environment - Java 1.6

public List<Attachment> download(List<Attachment> attachments)
{
  for(Attachment attachment : attachments) {
    attachment.setDownStatus("Failed");
    String destLocation = "C:\Users\attachments";
    try {
        String attUrl = attachment.getUrl();
        String fileName = attachment.getFileName();            
        URL url = new URL(attUrl);
        File fileLocation = new File(destLoc, fileName);
        FileUtils.copyURLToFile(url, fileLocation);
        if(fileLocation.exists()) {
           attachment.setDownStatus("Completed");
         }
       } catch(Exception e) {
          attachment.setDownStatus("Failed");
       } finally {
          attachment.setDestLocation(destLocation);
       }
   }
  return attachments;
}

I am downloading the file from provided URL (http://cdn.octafinance.com/wp-content/uploads/2015/07/google-hummingbird.jpg).

FileUtils.copyURLToFile(url, fileLocation);

The above code does its downloading job perfectly, without any issues.

My Problem:
If the list of attachments are more it will take more time, so I would like to make it an asynchronous or parallel process instead of downloading sequentially.


Solution

  • Actually, after carefully looking, Boris' code is faulty and will indeed not set some stuff sometimes. Here's a better version that fixes that:

    public List<Attachment> download(List<Attachment> attachments) {
      ExecutorService executorService = Executors.newCachedThreadPool();
      List<Future<Attachment>> futures = new ArrayList<Future<Attachment>>();
      for (final Attachment attachment : attachments) {
        futures.add(executorService.submit(new Callable<Attachment>() {
          @Override
          public Attachment call() throws Exception {
            return doDownload(attachment);
          }
        }));
      }
      for (Future<Attachment> future: futures) {
        try {
          future.get();
        } catch (Exception ex) {
          // Do something
        }
      }
      return attachments;
    }
    
    private Attachment doDownload(Attachment attachment) throws Exception {
      attachment.setDownStatus("Failed");
      attachment.setDestLocation("C:\\Users\\attachments");
      String attUrl = attachment.getUrl();
      String fileName = attachment.getFileName();
      URL url = new URL(attUrl);
      File fileLocation = new File(attachment.getDestLocation(), fileName);
      FileUtils.copyURLToFile(url, fileLocation);
      if (fileLocation.exists()) {
        attachment.setDownStatus("Completed");
      }
      return attachment;
    }
    

    However, this is absolutely not optimal given your structure of Attachment and how you use it. I did not fix that: I only answered the question as it was asked.