Very slow to list rasters with arcpy

Question

I'm using simple arcpy ListRasters functions and the script is very slow to execute.

I'm using them with a folder containing about 100 rasters (20 years of historical data for 5 different categories). Each raster is ~800Mo so ~80Go in total. I'm using wildcards to list them in 5 different lists:

import arcpy, os, sys
from arcpy import env
from arcpy import *
from arcpy.sa import *

arcpy.CheckOutExtension("Spatial")

hist_data_path = "D:/Data/GIS/hist_data"

arcpy.env.workspace = hist_data_path

hist_pop_urban = arcpy.ListRasters("*pop_urb*")
hist_pop_rural = arcpy.ListRasters("*pop_rur*")
hist_ppc_urban = arcpy.ListRasters("*ppc_urb*")
hist_ppc_rural = arcpy.ListRasters("*ppc_rur*")
hist_ww_int = arcpy.ListRasters("*ww_int*")

[...]

It takes about 10 minutes to list each bloc of 20 rasters... so ~50 min to list all the rasters... How is that possible? Do I miss something in the code? Is it because of the size of the rasters? Is there some "hidden" option or "trick" I could check? I'm using Win 7 64 on an i7 computer with 16Go RAM.

Thanks for any idea that could reduce this processing time..!

Answer 1

In my experience arcpy is generally slow, so I try to avoid it whenever possible. Of course, there may be some way to optimize the arcpy.ListRasters function, and I would love to hear about it if anyone knows about it.

Here is an out-of-the-box Python alternative to arcpy.ListRasters:

import os

directory = r"D:\Data\GIS\hist_data"
extension_list = [".tif", ".tiff"]
hist_pop_urban = []
hist_pop_rural = []
#etc.

for file in os.listdir(directory):
    for extension in extension_list:
        if file.endswith(extension):
            if "pop_urb" in file:
                hist_pop_urban.append(file)
            elif "pop_rur" in file:
                hist_pop_rural.append(file)
            #etc.

You could build extension_list based on the contents of this webpage and your knowledge of the particular file types you are dealing with:

http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//009t0000000q000000

Depending on the format of your rasters, each raster may comprise more than one file. If so, you would have to incorporate that into the code as well.

Good luck!

Tom