Java Folder size estimate

Question

We are building a java application that enables the user to import large datasets from a third party application. The data in the third party application is on the filesystem and distributed into a huge number of folders and small files (which a lot of users have on external disks). In order to protect the user, we want to warn him, if there is not enough disk space available to perform the import. However, to do so we have to calculate the disk space used by this huge lot of small files.

I tried using Apache IO and java.nio approaches to calculate the directory size. However, both methods take about 10 minutes with about 50GB of data on a FireWire disk.

This is too long as this task is a pure safety measure and most of the time, we arrive at the solution that there is enough space available.

Is there some method that can produce a fast, intelligent raw estimate about the space consumed by the directory?

Answer 1

I would also be interested in knowing if there is a bit that holds the size of the disk.

Meanwhile here is my solution - with a bit of fancy text processing of the output you can get the size - it takes abour 6 min for 70G again probably not what you are looking for but it could half your time

long tm=System.currentTimeMillis();
try {
  String cmd="cmd /c dir c:\\  /s  ";
  execute(cmd, false);
}
catch (Exception ex) { }
  System.out.println((System.currentTimeMillis()-tm));


public String execute(String cmd, boolean getoutput) {
String output=null;
  try {
    Runtime rt = Runtime.getRuntime();
    Process pr=rt.exec(cmd);
    StreamGobbler errorGobbler=new StreamGobbler(pr.getErrorStream(), "ERROR", getoutput);
    errorGobbler.start();
    StreamGobbler inputGobbler=new StreamGobbler(pr.getInputStream(), "INPUT", getoutput);
    inputGobbler.start();
    int exitVal=pr.waitFor();
    System.out.println("ExitValue: " + exitVal);
    output=""+errorGobbler.output;
    output+=inputGobbler.output;
  }
  catch(Throwable t) { t.printStackTrace(); }
  return output;
}


import java.util.*;
import java.io.*;

public class StreamGobbler extends Thread {
boolean redirect=false;
InputStream is;
OutputStream os;
String type, output="";

StreamGobbler(InputStream is, String type) {
    this.is = is;
    this.type = type;
}

StreamGobbler(InputStream is, String type, boolean redirect) {
    this.is = is;
    this.type = type;
    this.redirect=redirect;
}

StreamGobbler(OutputStream os, String type) {
    this.os = os;
    this.type = type;
}

StreamGobbler(OutputStream is, String type, boolean redirect) {
    this.os = os;
    this.type = type;
    this.redirect=redirect;
}

 public void run() {
    try
    {
        if(type.equals("OUTPUT")) {
        }
        else {
        InputStreamReader isr = new InputStreamReader(is);
        BufferedReader br = new BufferedReader(isr);
        String line=null;
        int i=0;
        while ( (line = br.readLine()) != null) {
            if(redirect) output+=line;
            else System.out.println("line "+i+" "+type + ">" + line);
            i++;
        }
        }
        } catch (IOException ioe)
          {
            ioe.printStackTrace();
          }
}   

}