I am trying to run a python-script via CGI but I get Internal server error
error log according to apache2:
[Mon Dec 21 10:43:19.073771 2020] [cgi:error] [pid 7531] [client ::1:50038] AH01215: File "/usr/lib/cgi-bin/pytest.py", line 5, in <module>: /usr/lib/cgi-bin/pytest.py
[Mon Dec 21 10:43:19.073809 2020] [cgi:error] [pid 7531] [client ::1:50038] AH01215: from bs4 import BeautifulSoup : /usr/lib/cgi-bin/pytest.py
[Mon Dec 21 10:43:19.073832 2020] [cgi:error] [pid 7531] [client ::1:50038] AH01215: ModuleNotFoundError: No module named 'bs4': /usr/lib/cgi-bin/pytest.py
It's all about a pretty simple scraper-script that I want to run via CGI. I would emphasize that running this script normally through terminal is no problem, its just a problem using CGI
Is it possible to solve this, or is CGI not fitted for these kind of scripts?
code:
#! /usr/bin/python3
# enable debugging
import cgitb
from bs4 import BeautifulSoup
from urllib.request import urlopen, Request
import os
cgitb.enable()
print ("Content-type: text/html\r\n\r\n")
print("hello")
url = "https://www.xxxxxxxxxxxxx"
soup = BeautifulSoup(urlopen(url).read())
...
If it is not the same Python executable, that's a problem, You are therefore running an installation of Python that doesn't have BeautifulSoup installed. You need to use the same executable or create a virtual environment from this executable, install the required modules in the virtual environment and point the shebang to the Python executable within the virtual environment.
For example, to use the /usr/bin/python3 executable with a virtual environment that has your modules:
cd /home/usr/mydir
/usr/bin/python3 -m venv my_venv
cd my_venv/bin
. activate
pip install bs4
deactivate
Then change shebang to point to:
#!/home/usr/mydir/my_venv/bin/python3