Compress output raster and parallelize gdalwarp from R

I would like to include -co options to compress output raster using gdalwarp from gdalUtilities in R.

I have tried some options (commented in the code), but I have not been successful in generating the compressed raster.

gdalUtilities::gdalwarp(srcfile = paste0(source_path,"/mask_30.tif"),
                        dstfile = paste0(writing_path,"/mask_30_gdalwarp.tif"),
                        cutline = paste0(source_path,"/amazon.shp"),
                        crop_to_cutline = TRUE,
                        multi = TRUE,
                        wo = "NUM_THREADS = 32",
                        co = "COMPRESS = DEFLATE")
                        # co = c("COMPRESS = DEFLATE","ZLEVEL = 9"))
                        # co COMPRESS = DEFLATE,
                        # co ZLEVEL = 9),
                        # co = "COMPRESS = DEFLATE",
                        # co = ZLEVEL = 9")

Additionally, I would like to use multithread warping implementation. I am including-multi and -wo "NUM_THREADS = 16" (my computer has 32 cores) options, but I also have not been able to decrease the runtime vs. the default -multi option, which uses two cores by default.

Any suggestions for compression and parallelization?

Many thanks in advance.

Solution

1 - COMPRESSION

Please find the solution for the problem of file compression. To be honest, I have already been confronted with the same problem as you and, at the time, I was racking my brains... to finally find the solution which is quite simple (once we know it!): you must not put any spaces (i.e. "COMPRESS=DEFLATE" and not "COMPRESS = DEFLATE")

So, please find below a small reprex.

Reprex

library(gdalUtilities)
library(stars)        # Loaded just to have a '.tif' image for the reprex


# Import a '.tif' image from the 'stars' library
tif <- read_stars(system.file("tif/L7_ETMs.tif", package = "stars"))

# Write the image to disk (in your working directory)
write_stars(tif, "image.tif")

# Size of the image on disk (in bytes)
file.size("image.tif")
#> [1] 2950880

# Compress the image
gdalUtilities::gdalwarp(srcfile = "image.tif",
                        dstfile = "image_gdalwarp.tif",
                        co = "COMPRESS=DEFLATE")

# Size of the compressed image on disk (in bytes)
file.size("image_gdalwarp.tif")
#> [1] 937920                # The image has been successfully compressed.

As @MarkAdler said, there is not much difference between the default compression level (i.e. 6) and level 9. That said, please find below how you should write the code to be able to apply the desired compression level (i.e. still without spaces and in a list):

gdalUtilities::gdalwarp(srcfile = "image.tif",
                        dstfile = "image_gdalwarp_Z9.tif",
                        co = list("COMPRESS=DEFLATE", "ZLEVEL=9"))

file.size("image_gdalwarp_Z9.tif")
#> [1] 901542

^{Created on 2022-02-09 by the reprex package (v2.0.1)}

2 - PARALLELIZATION

For the problem of parallelization on the cores of the processor, you should not use multi = TRUE. Only the argument wo = "NUM_THREADS=4" (always without spaces ;-)) is enough.

Just a clarification, I guess you are confusing the RAM and the number of cores. Usually computers are equipped with a 4 or 8 cores processor. The 32 that you indicate in your code refers to the 32 gigas of RAM that your computer probably has.

Reprex

library(gdalUtilities)
library(stars)

tif <- read_stars(system.file("tif/L7_ETMs.tif", package = "stars"))

write_stars(tif, "image.tif")


file.size("image.tif")
#> [1] 2950880

gdalUtilities::gdalwarp(srcfile = "image.tif",
                        dstfile = "image_gdalwarp_Z9_parallel.tif",
                        co = list("COMPRESS=DEFLATE", "ZLEVEL=9"),
                        wo = "NUM_THREADS=4") # Replace '4' by '8' if your processor has 8 cores

file.size("image_gdalwarp_Z9_parallel.tif")
#> [1] 901542

^{Created on 2022-02-09 by the reprex package (v2.0.1)}