Search code examples
pythonnumpyapple-m1arm64

Python do not work when executed from Java on M1 Mac


I have a bash script that runs a python script:

#!/bin/bash
restest-env/bin/python3 script.py $1 $2 $3

When executed from terminal, everything works fine. Instead, when executed from a Java application with:

ProcessBuilder pb = new ProcessBuilder(command, String.join(" ",commandArgs));
Process proc = pb.start();
proc.getOutputStream();
String stdout = IOUtils.toString(proc.getInputStream(), Charset.defaultCharset());
String stderr = IOUtils.toString(proc.getErrorStream(), Charset.defaultCharset());
proc.waitFor();

I get this M1 chip-related error:

Traceback (most recent call last):
  File "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dlopen(/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 0x0002): tried: '/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ml/python-scripts/al_predictor.py", line 2, in <module>
    import numpy as np
  File "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/__init__.py", line 140, in <module>
    from . import core
  File "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.8 from "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/bin/python3"
  * The NumPy version is: "1.23.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 0x0002): tried: '/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

2022-07-18 11:21:14 INFO  stdout:36 - Traceback (most recent call last):
  File "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dlopen(/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 0x0002): tried: '/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ml/python-scripts/al_predictor.py", line 2, in <module>
    import numpy as np
  File "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/__init__.py", line 140, in <module>
    from . import core
  File "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.8 from "/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/bin/python3"
  * The NumPy version is: "1.23.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 0x0002): tried: '/Users/giulianomirabella/Desktop/RESTest/ml/restest-env/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

So the strange thing is that python works fine while executed directly from terminal, but fails when executed from Java. I can not use conda. Any ideas?


Solution

  • I had the same issue where I was trying to use Java ProcessBuilder to run a terminal command to run a python project.

    My command ran fine in the terminal but was not working when run from the Java program. When running command 'uname -p' I could see when the terminal was open manually I got 'arm' but when Java ran it it was 'x86_64 i386'. Basically java was running as an x86_64 process but my python install was arm so it failed.

    To solve the issue I simply installed java jdk 18.0.2 and run my java program using this jdk. It worked after this