Python, Chaquopy, ImportError on Android when using tensorflow due to lack of semaphores

Question

I'm running keras with tensorflow on Android using Chaquopy. When I call predict() on my model, I get an exception with the following stack trace:

AndroidRuntime: Process: com.example.android.camera2basic, PID: 10113
AndroidRuntime: com.chaquo.python.PyException: ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
AndroidRuntime:     at <python>.multiprocessing.synchronize.<module>(synchronize.py:30)
AndroidRuntime:     at <python>.zipimport.load_module(<frozen zipimport>:259)
AndroidRuntime:     at <python>.java.chaquopy.import_override(import.pxi:60)
AndroidRuntime:     at <python>.multiprocessing.context.Lock(context.py:67)
AndroidRuntime:     at <python>.multiprocessing.queues.__init__(queues.py:336)
AndroidRuntime:     at <python>.multiprocessing.context.SimpleQueue(context.py:113)
AndroidRuntime:     at <python>.multiprocessing.pool.__init__(pool.py:196)
AndroidRuntime:     at <python>.multiprocessing.pool.__init__(pool.py:922)
AndroidRuntime:     at <python>.tensorflow.python.keras.engine.training_utils.get_copy_pool(training_utils.py:210)
AndroidRuntime:     at <python>.tensorflow.python.keras.engine.training_utils.__init__(training_utils.py:242)
AndroidRuntime:     at <python>.tensorflow.python.keras.engine.training_utils.create(training_utils.py:335)
AndroidRuntime:     at <python>.tensorflow.python.keras.engine.training_v2.run_one_epoch(training_v2.py:171)
AndroidRuntime:     at <python>.tensorflow.python.keras.engine.training_v2._model_iteration(training_v2.py:464)
AndroidRuntime:     at <python>.tensorflow.python.keras.engine.training_v2.predict(training_v2.py:495)
AndroidRuntime:     at <python>.tensorflow.python.keras.engine.training.predict(training.py:1004)

My understanding is that Android does not support semaphores.

Does anyone have any sort of workaround for this?

Answer 1

Keras is actually trying to use a simple thread pool, but it looks like the standard library is pulling in some inter-process synchronization code anyway. I guess this hasn't been noticed because all the major platforms support semaphores.

To work around it, run the following code before you use Keras:

    import multiprocessing
    import threading
    def threading_func(name):
        def f(self, *args, **kwargs):
            return getattr(threading, name)(*args, **kwargs)
        f.__name__ = f.__qualname__ = name
        return f

    ctx = multiprocessing.get_context()
    for name in ["Lock", "RLock", "Condition", "Semaphore", "BoundedSemaphore",
                 "Event", "Barrier"]:
        setattr(type(ctx), name, threading_func(name))
        setattr(multiprocessing, name, getattr(ctx, name))

Please let me know in a comment whether this works, as I'll probably incorporate it into the next version of Chaquopy.