Search code examples
pythonpython-2.7multiprocessingarcpy

Multiprocessing arcpy python run from batch file


So I'm trying to add multiprocessing capability to a python script that's run via a batch file. I keep getting a traceback type error when run and can't seem to trouble shoot it. The most I can puzzle out of the error is that it says Medcen function does not exist. But I thought that was the purpose of the def (see below).

EDIT

Code has been edited to reflect current state after suggestions.

So I see now that part of my issue is I'm trying to convert a for loop to a multiprocess. Where I could do an incrementing variable via x = x+1 to modify the output filenames I now need to find another way to do that. But that's not the entire problem. I've attempted using enumerate but that doesn't work either.

Batch file code:

C:\PathToArcPython2.7-64bit C:\PathtoScript3

Script 3 Code multiprocessing attempt:

import arcpy
import os
import fnmatch
import sys
import multiprocessing
#==============================================================================     
working_dir = r'E:\PathToDir'
output_dir = os.path.join(working_dir, 'Results')
if not os.path.isdir(output_dir):
    os.mkdir(output_dir)
#============================================================================== 
global input_files5
global input_files6
input_files5 = []
input_files6 = []
#==============================================================================
for r, d, f in os.walk(output_dir):
    for inFile in fnmatch.filter(f, '*Processed.shp'):
        input_files5.append(os.path.join(r, inFile))

global num
num = enumerate([0,1])

def Script(file):
    name = output_dir + "\\" + "Prefix" + str(num) + "Script.shp"
    arcpy.Tool(file, name,"","", "Field1;Field2") 
    input_files6.append(name)
    print "Script " + str(num)

def MCprocess():
    pool = multiprocessing.Pool(processes=4)
    pool.map(Script, input_files5)

if __name__ == '__main__':
    working_dir = r'E:\PathToDir'
    output_dir = os.path.join(working_dir, 'Results')
    input_files5 = []
    input_files6 = []
    for r, d, f in os.walk(output_dir):
        for inFile in fnmatch.filter(f, '*Processed.shp'):
            input_files5.append(os.path.join(r, inFile))
    MCprocess()

StackTraceError

Traceback (most recent call last):
  File "Path\To\Script.py", line 55, in <module>
    MCprocess()
  File "Path\To\Script.py", line 41, in MCprocess
    pool.map(Script, input_files5)
  File "C:\Python27\ArcGISx6410.5\lib\multiprocessing\pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "C:\Python27\ArcGISx6410.5\lib\multiprocessing\pool.py", line 567, in get
    raise self._value
ExecuteError: ERROR 000210: Cannot create output 
Path\To\File\Prefix_<enumerate object at 0x0000000012CCB678>Suffix.shp
Failed to execute (Script).

Solution

  • The issue was trying to iterate in the same as with a for loop. Instead I decided to do it with the original file names which are already unique at this stage of the processing. Relevant code below.

    def Medcen(file):
    
        filename = str(file)
        filename = filename.split('\\')
        filename = filename[-1]
        filename = filename.strip('Processed.shp')
    
        name = output_dir + "\\" + str(filename) + "MedCen.shp"
    
        arcpy.MedianCenter_stats(file, name,"","", "Field1;Field2") 
        input_files6.append(name)
        print "Med Cen " + str(filename)