Python: memmap list of objects become 'None' type inside joblib parallel

Question

I am doing the following:

1. I have a list of tensorflow DNN layers. nn.append(tf.layers.dense(...))
2. Each of the above list is appended to a list of np.memmap objects. nnList[i] = nn
3. I can access the memmap list and retrieve the tensors. But when try to access the tensors inside joblib.parallel it returns 'None' type object. However, the length of the memmap list is correct inside joblib.parallel.

I have attached a sample code below.

    import os
    import tempfile
    import numpy as np
    import tensorflow as tf
    from joblib import Parallel, delayed, load, dump

    tmpFolder = tempfile.mkdtemp()
    __nnFile = os.path.join(tmpFolder, 'nn.mmap')
    nnList = np.memmap(__nnFile, dtype=object, mode='w+', shape=(5))

    def main():
        for i in range(5):
            nn = []
            input = tf.placeholder(dtype=tf.float32, shape=(1, 8))
            nn.append(tf.layers.dense(inputs=input, units=8, activation=tf.sigmoid,  
                                        trainable=False))
            nn.append(tf.layers.dense(inputs=nn[0], units=2, activation=tf.sigmoid,  
                                        trainable=False))

            nnList[i] = nn

        print('nnList: ' + str(len(nnList)))
        for i in range(5):
            nn = nnList[i]
            print(nn)
            print(nn[-1])
            print('---------------------------  ' + str(i))

        with Parallel(n_jobs = -1) as parallel:
            parallel(delayed(func1)(i) for i in range(5))

    def func1(i):
        print('nnList: ' + str(len(nnList)))
        for x in range(5):
            nn = nnList[x]
            print(nn)
            print('---------------------------  ' + str(x))

    if __name__ == '__main__':
        main()

The above code gives this output. Note the length of the arrays and how the tensors become None.

    nnList: 5
    [<tf.Tensor 'dense/Sigmoid:0' shape=(1, 8) dtype=float32>, <tf.Tensor 'dense_1/Sigmoid:0' shape=(1, 2) dtype=float32>]
    Tensor("dense_1/Sigmoid:0", shape=(1, 2), dtype=float32)
    ---------------------------  0
    [<tf.Tensor 'dense_2/Sigmoid:0' shape=(1, 8) dtype=float32>, <tf.Tensor 'dense_3/Sigmoid:0' shape=(1, 2) dtype=float32>]
    Tensor("dense_3/Sigmoid:0", shape=(1, 2), dtype=float32)
    ---------------------------  1
    [<tf.Tensor 'dense_4/Sigmoid:0' shape=(1, 8) dtype=float32>, <tf.Tensor 'dense_5/Sigmoid:0' shape=(1, 2) dtype=float32>]
    Tensor("dense_5/Sigmoid:0", shape=(1, 2), dtype=float32)
    ---------------------------  2
    [<tf.Tensor 'dense_6/Sigmoid:0' shape=(1, 8) dtype=float32>, <tf.Tensor 'dense_7/Sigmoid:0' shape=(1, 2) dtype=float32>]
    Tensor("dense_7/Sigmoid:0", shape=(1, 2), dtype=float32)
    ---------------------------  3
    [<tf.Tensor 'dense_8/Sigmoid:0' shape=(1, 8) dtype=float32>, <tf.Tensor 'dense_9/Sigmoid:0' shape=(1, 2) dtype=float32>]
    Tensor("dense_9/Sigmoid:0", shape=(1, 2), dtype=float32)
    ---------------------------  4
    nnList: 5
    None
    ---------------------------  0
    None
    ---------------------------  1
    None
    ---------------------------  2
    None
    ---------------------------  3
    None
    ---------------------------  4

How can I access the tensors inside joblib.parallel? Please help.

Answer 1

Found the issue back then. Hope it helps someone in the future.

The None problem had nothing to do with the tensors. I was using the joblib.Parallel function the wrong way.

One should pass the variable to delayed to be accessible to the forked processes (how did I overlook that in the documentation!). The correct way:

with Parallel(n_jobs = -1) as parallel:
    parallel(delayed(func1)(i, WHATEVER_VARIABLE_I_WANT) for i in range(5))