cannot append results to lists on multiprocessing

Question

This below code will generate face encodings using multiprocessing , i can able to print the encoding but the problem is the knownEncodings ,knownNames ,no_faces ,error_in_image all are empty after the execution. I know its due to multiprocessing , but not sure how to mitigate this.

import face_recognition
from imutils import paths
from multiprocessing import Pool
import pickle
import cv2
import os,sys,time

print("[INFO] quantifying faces...")

img_folder_path=sys.argv[1]

image_paths = list(paths.list_images(img_folder_path))

knownEncodings = []
knownNames = []
no_faces = []
error_in_image =[]

def create_encoding(imagePath):
    print("[INFO] processing image...")
    name = imagePath.split(os.path.sep)[-1]
    image = cv2.imread(imagePath)
    if image is None:
        return
    rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

    # detect the (x, y)-coordinates of the bounding boxes
    # corresponding to each face in the input image
    boxes = face_recognition.face_locations(rgb)

    # compute the facial embedding for the face
    if len(boxes) != 0:
        boxes = list(boxes[0])
        encodings = face_recognition.face_encodings(image, [boxes])
        for encoding in encodings:  
            knownEncodings.append(encoding)
            knownNames.append(name)
        
    else:
        print("no face found" ,image_paths )
        no_faces.append(image_paths )



# loop over the image paths with multiprocessing
start_time = time.time()

with Pool(8) as pool:
    pool.map(create_encoding, image_paths )


end_time = time.time()
print(end_time - start_time)

# dump the facial encodings + names to disk
print("[INFO] serializing encodings...")
data = {"encodings": knownEncodings, "names": knownNames, "no_faces":no_faces,"error_in_image":error_in_image}

f_name = img_folder_path.replace("/","-")
print(f_name)
f = open(f"encodings_{f_name}.pickle", "wb")
f.write(pickle.dumps(data))
f.close()

Answer 1

Presumably your are running under an OS such as Linux that uses fork to create new processes or else you would have needed to put the code that creates new processes within an if __name__ == '__main__': block. But you really should tag multiprocessing questions with the actual platform you are running. Your problem (in any case) is that your global lists, such as knownEncodings, is not sharable across processes. Each process (in the case of Linux) inherits the address space of the main process as it looked at the time it was created but once it makes a modification to a variable, a copy of that storage is made ("copy on write" semantics). In essence each process has its own copy of global variables.

There are, however, numerous ways to share data across processes. The simplest solution, that is, the one that requires the fewest changes to your program, is to use "managed lists" created by the multiprocessing.managers,SyncManager class, an instance of which can be created by calling multiprocessing.Manager() (this also results in a process being created). The changes you need are:

Assuming that you are running under a platform that uses fork, such as Linux, replace your global definitions of the four lists that you have at the top of the program with:

manager = Manager()
knownEncodings = manager.list()
knownNames = manager.list()
no_faces = manager.list()
error_in_image = manager.list()