While running models/research/object_detection/model_main_tf2.py from tensorflow/models (or just python -c "from object_detection import model_lib_v2") I get:
2021-05-24 11:09:02.622828: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Segmentation fault (core dumped)
I used this guide to setup the object detection API and i had no issues, the python object_detection/builders/model_builder_tf2_test.py works as well.
nvidia-smi output:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.73.01 Driver Version: 460.73.01 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1050 Off | 00000000:01:00.0 Off | N/A |
| N/A 55C P0 N/A / N/A | 248MiB / 4042MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
Python version: 3.8.5
Tensorflow version: 2.5.0
OS: Ubuntu 20.04.2 LTS (Focal Fossa)
EDIT: Add debug with Python3's faulthandler:
2021-05-28 12:48:41.761418: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Fatal Python error: Segmentation fault
Current thread 0x00007fd3e6de8740 (most recent call first):
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 1101 in create_module
File "<frozen importlib._bootstrap>", line 556 in module_from_spec
File "<frozen importlib._bootstrap>", line 657 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
File "/home/alessandro/.local/lib/python3.8/site-packages/PIL/ImageFont.py", line 48 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 783 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "/usr/local/lib/python3.8/dist-packages/object_detection/utils/visualization_utils.py", line 35 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 783 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
File "/usr/local/lib/python3.8/dist-packages/object_detection/eval_util.py", line 42 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 783 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
File "/usr/local/lib/python3.8/dist-packages/object_detection/model_lib_v2.py", line 29 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 783 in exec_module
File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 991 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
File "<stdin>", line 1 in <module>
Segmentation fault (core dumped)
I managed to resolve by downgrading Pillow to 7.0.0, downgrading numpy to 1.19.5 (which is the latest version still compatible with tensorflow 2.5.0 at the moment) and downgrading pycocotools to 2.0.0.