So I'm trying to add multiprocessing capability to a python script that's run via a batch file. I keep getting a traceback type error when run and can't seem to trouble shoot it. The most I can puzzle out of the error is that it says Medcen function does not exist. But I thought that was the purpose of the def (see below).
EDIT
Code has been edited to reflect current state after suggestions.
So I see now that part of my issue is I'm trying to convert a for loop to a multiprocess. Where I could do an incrementing variable via x = x+1 to modify the output filenames I now need to find another way to do that. But that's not the entire problem. I've attempted using enumerate but that doesn't work either.
Batch file code:
C:\PathToArcPython2.7-64bit C:\PathtoScript3
Script 3 Code multiprocessing attempt:
import arcpy
import os
import fnmatch
import sys
import multiprocessing
#==============================================================================
working_dir = r'E:\PathToDir'
output_dir = os.path.join(working_dir, 'Results')
if not os.path.isdir(output_dir):
os.mkdir(output_dir)
#==============================================================================
global input_files5
global input_files6
input_files5 = []
input_files6 = []
#==============================================================================
for r, d, f in os.walk(output_dir):
for inFile in fnmatch.filter(f, '*Processed.shp'):
input_files5.append(os.path.join(r, inFile))
global num
num = enumerate([0,1])
def Script(file):
name = output_dir + "\\" + "Prefix" + str(num) + "Script.shp"
arcpy.Tool(file, name,"","", "Field1;Field2")
input_files6.append(name)
print "Script " + str(num)
def MCprocess():
pool = multiprocessing.Pool(processes=4)
pool.map(Script, input_files5)
if __name__ == '__main__':
working_dir = r'E:\PathToDir'
output_dir = os.path.join(working_dir, 'Results')
input_files5 = []
input_files6 = []
for r, d, f in os.walk(output_dir):
for inFile in fnmatch.filter(f, '*Processed.shp'):
input_files5.append(os.path.join(r, inFile))
MCprocess()
StackTraceError
Traceback (most recent call last):
File "Path\To\Script.py", line 55, in <module>
MCprocess()
File "Path\To\Script.py", line 41, in MCprocess
pool.map(Script, input_files5)
File "C:\Python27\ArcGISx6410.5\lib\multiprocessing\pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File "C:\Python27\ArcGISx6410.5\lib\multiprocessing\pool.py", line 567, in get
raise self._value
ExecuteError: ERROR 000210: Cannot create output
Path\To\File\Prefix_<enumerate object at 0x0000000012CCB678>Suffix.shp
Failed to execute (Script).
The issue was trying to iterate in the same as with a for loop. Instead I decided to do it with the original file names which are already unique at this stage of the processing. Relevant code below.
def Medcen(file):
filename = str(file)
filename = filename.split('\\')
filename = filename[-1]
filename = filename.strip('Processed.shp')
name = output_dir + "\\" + str(filename) + "MedCen.shp"
arcpy.MedianCenter_stats(file, name,"","", "Field1;Field2")
input_files6.append(name)
print "Med Cen " + str(filename)