I'm running a python script in the command line with python3 CTGAN_noscale.py --database_name CTGAN_noshift and receive the following error (with faulthandler):
Fatal Python error: Segmentation fault
Current thread 0x00007f57e97fe700 (most recent call first):
<no Python frame>
Thread 0x00007f593db07740 (most recent call first):
File "/usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py", line 145 in backward
File "/usr/local/lib/python3.8/dist-packages/torch/tensor.py", line 245 in backward
File "/usr/local/lib/python3.8/dist-packages/ctgan/synthesizers/ctgan.py", line 374 in fit
File "CTGAN_noscale.py", line 140 in objective
File "CTGAN_noscale.py", line 162 in <lambda>
File "/usr/local/lib/python3.8/dist-packages/optuna/_optimize.py", line 216 in _run_trial
File "/usr/local/lib/python3.8/dist-packages/optuna/_optimize.py", line 162 in _optimize_sequential
File "/usr/local/lib/python3.8/dist-packages/optuna/_optimize.py", line 65 in _optimize
File "/usr/local/lib/python3.8/dist-packages/optuna/study.py", line 401 in optimize
File "CTGAN_noscale.py", line 162 in run_CTGAN
File "CTGAN_noscale.py", line 210 in <module>
Segmentation fault (core dumped)
It seems that the problem is somehow with optuna. The weird part is that everything worked fine on another server, after changing the server it crashed like this.
Update
I found out that the the problem doesn't occur when I don't use a docker container OR use a docker container without GPU.
I solved the problem by rebuilding a new image and derived a container from this image. In this container somehow the error didn't appear anymore.