Search code examples
javacompressiontruezip

TrueZip compression taking too much time


I am using TrueZip for compression. Here is what my code looks like

  public String compress() throws IOException {
    if (logLocations.isEmpty()) {
      throw new IllegalStateException("no logs provided to compress");
    }

    removeDestinationIfExists(desiredArchive);

    final TFile destinationArchive = new TFile(desiredArchive + "/diagnostics");
    for (final String logLocation : logLocations) {
      final TFile log = new TFile(logLocation);
      if (!log.exists()) {
        LOGGER.debug("{} does not exist, ignoring.");
        continue;
      }
      if (log.isDirectory()) {
        log.cp_r(destinationArchive);
      } else {
        final String newLogLocation =
            new TFile(destinationArchive.getAbsolutePath()) + SLASH +
            getLogNameFromPath(logLocation);
        log.cp(new TFile(newLogLocation));
      }
    }
    return destinationArchive.getEnclArchive().getAbsolutePath();
  }

and my test

@Test
  public void testBenchMarkWithHprof() throws IOException {
    final FileWriter logLocations;
    String logLocationPath = "/Users/harit/Downloads/tmp/logLocations.txt";
    {
      logLocations = new FileWriter(logLocationPath);
      logLocations.write("Test3");
      logLocations.write("\n");
      logLocations.close();
    }
    final LPLogCompressor compressor = new LPLogCompressor("/Users/harit/Downloads/tmp",
                                                           new File(logLocationPath),
                                                           "/Users/harit/Downloads/tmp/TestOut");
    final long startTime = System.currentTimeMillis();
    compressor.compress();
    System.out.println("Time taken (msec): " + (System.currentTimeMillis() - startTime));
  }

and my data directory Test3 looks like

Test3/
      java_pid1748.hprof

The file size is 2.83GB When I ran the test, it took over 22 minutes.
However when I compress the same file using Native OSX compress (right click -> compress), it takes only 2 minutes

Why there is so much of difference?

Thanks

UPDATE

Based on @Satnam recommendation, I attached a debugger to see whats going on and this is what I find

enter image description here enter image description here

None of the TrueZip Threads are running? really? Apologies I am using profiler for the first time


Solution

  • The reason in this case was using default Deflater which is Deflater.BEST_COMPRESSION.

    I override the ZipDriver class to over the level as

    import de.schlichtherle.truezip.fs.archive.zip.ZipDriver;
    import de.schlichtherle.truezip.socket.IOPoolProvider;
    
    import java.util.zip.Deflater;
    
    public class OverrideZipDriver extends ZipDriver {
    
      public OverrideZipDriver(final IOPoolProvider ioPoolProvider) {
        super(ioPoolProvider);
      }
    
      @Override
      public int getLevel() {
        return Deflater.DEFAULT_COMPRESSION;
      }
    }
    

    and then in my Compressor class, I did

    public LPLogCompressor(final String logProcessorInstallPath, final File logLocationsSource,
                             final String desiredArchive) throws IOException {
        this.desiredArchive = desiredArchive + DOT + getDateTimeStampFormat() + ZIP;
        logLocations = getLogLocations(logProcessorInstallPath, logLocationsSource);
        enableLogCompression();
      }
    
      private static void enableLogCompression() {
        TConfig.get().setArchiveDetector(
            new TArchiveDetector(TArchiveDetector.NULL, new Object[][]{
                {"zip", new OverrideZipDriver(IOPoolLocator.SINGLETON)},}));
        TConfig.push();
      }
    

    You can read the thread here