my code is running on a 32-bit JVM (JRE v1.6) on Windows 2008 Server (64-bit) with 128 GB of RAM and 64 cores. however, the maximum heap space i can ever specify is 1.5 GB. my code looks the following.
int numThreads = Runtime.getRuntime.availableProcessors();
List<Callable<Long>> tasks = new ArrayList<Callable<Long>>();
File dir = new File("/path/to/data");
File[] dataFiles = dir.listFiles();
for(File dataFile : dataFiles) {
MyTask task = new MyTask(dataFile);
tasks.add(task);
}
ExecutorService executor = Executors.newFixedThreadPoll(numThreads);
List<Future<Long>> results = executor.invokeAll(tasks);
long total = 0L;
for(Future<Long> result : results) {
total += result.get();
}
System.out.println("total = " + total);
executor.shutdown();
this code throws an OutOfMemoryError. what i have done is changed the number of threads to be something smaller.
int numThreads = Runtime.getRuntime.availableProcessors();
if(numThreads < 1 || numThreads > 4) {
numThreads = 4;
}
this revised code hasn't yet thrown an OutOfMemoryError, but, it is disappointing to me because there are so much resources (RAM and CPU resources) not being used. how can i try to maximize the resource usage in my environment?
most importantly, i'd like some feedback on a workaround regarding the 1.5 GB maximum heap space limitation. note, the Callable<Long> tasks are embarassingly parallel.
i have thought about creating a DOS bat file to iterate over my input files and then simply call
java -cp %CP% -Xms1024m -Xmx1536m net.analysis.MyProg %1
but this seems kind of quirky/kludgy (now i have to have logic in DOS bat to determine how many processes to create, and wait for those processes to finish before spawning new ones).
any help is appreciated.
Options: