I'm trying to install a new server, and I can't run Django with nginx and uwsgi. I receive an error "502 Bad Gateway" and there are messages on the error log which I don't understand:
2019/07/20 10:50:44 [error] 2590#2590: *1 upstream prematurely closed connection while reading response header from upstream, client: 79.183.208.33, server: *.speedy.net.3.speedy-technologies.com, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/run/uwsgi/app/speedy_net/socket:", host: "3.speedy-technologies.com"
I have 4 websites and here is the main (default) configuration file:
server {
listen [::]:80 default_server;
listen 80 default_server;
server_name *.speedy.net.3.speedy-technologies.com speedy.net.3.speedy-technologies.com;
access_log /var/log/nginx/speedy-net.access.log;
error_log /var/log/nginx/speedy-net.error.log;
client_max_body_size 50M;
root /home/ubuntu/speedy-net/speedy/net/static_serve/root;
try_files $uri @uwsgi;
location @uwsgi {
include uwsgi_params;
uwsgi_pass unix:/run/uwsgi/app/speedy_net/socket;
}
location /admin/ {
auth_basic "admin site";
auth_basic_user_file /etc/nginx/htpasswd_admin;
include uwsgi_params;
uwsgi_pass unix:/run/uwsgi/app/speedy_net/socket;
}
location /static {
alias /home/ubuntu/speedy-net/speedy/net/static_serve;
access_log off;
# expires max;
gzip on;
gzip_min_length 1100;
gzip_buffers 4 32k;
gzip_types text/css text/javascript text/xml text/plain text/x-component application/javascript application/x-javascript application/json application/xml application/rss+xml font/truetype application/x-font-ttf font/opentype application/vnd.ms-fontobject image/svg+xml;
gzip_vary on;
gzip_comp_level 6;
}
}
And:
[uwsgi]
project = net
chdir = /home/ubuntu/speedy-net
home = %(chdir)/env
module = speedy.%(project).wsgi:application
plugins = python3
master = true
processes = 4
chmod-socket = 666
vacuum = true
uid = ubuntu
gid = ubuntu
touch-reload = /run/uwsgi/app/speedy_%(project)/reload
I tried to test with sudo nginx -t but I received a message that my configuration is OK. What is the problem?
Edit:
The old server (which is working):
$ find /run/uwsgi/ -ls
489 0 drwxr-xr-x 3 root root 60 Jul 20 08:41 /run/uwsgi/
490 0 drwxr-xr-x 6 root root 120 Jul 20 08:41 /run/uwsgi/app
500 0 drwxr-xr-x 2 ubuntu ubuntu 100 Jul 20 08:42 /run/uwsgi/app/speedy_net
523 0 -rw-rw-r-- 1 ubuntu ubuntu 0 Jul 20 08:43 /run/uwsgi/app/speedy_net/reload
501 0 srw-rw-rw- 1 ubuntu ubuntu 0 Jul 20 08:41 /run/uwsgi/app/speedy_net/socket
457 4 -rw-r--r-- 1 root root 5 Jul 20 08:41 /run/uwsgi/app/speedy_net/pid
498 0 drwxr-xr-x 2 ubuntu ubuntu 100 Jul 20 08:42 /run/uwsgi/app/speedy_match
524 0 -rw-rw-r-- 1 ubuntu ubuntu 0 Jul 20 08:43 /run/uwsgi/app/speedy_match/reload
499 0 srw-rw-rw- 1 ubuntu ubuntu 0 Jul 20 08:41 /run/uwsgi/app/speedy_match/socket
413 4 -rw-r--r-- 1 root root 5 Jul 20 08:41 /run/uwsgi/app/speedy_match/pid
495 0 drwxr-xr-x 2 ubuntu ubuntu 100 Jul 20 08:43 /run/uwsgi/app/speedy_mail
528 0 -rw-rw-r-- 1 ubuntu ubuntu 0 Jul 20 08:43 /run/uwsgi/app/speedy_mail/reload
497 0 srw-rw-rw- 1 ubuntu ubuntu 0 Jul 20 08:41 /run/uwsgi/app/speedy_mail/socket
496 4 -rw-r--r-- 1 root root 5 Jul 20 08:41 /run/uwsgi/app/speedy_mail/pid
491 0 drwxr-xr-x 2 ubuntu ubuntu 100 Jul 20 08:42 /run/uwsgi/app/speedy_composer
527 0 -rw-rw-r-- 1 ubuntu ubuntu 0 Jul 20 08:43 /run/uwsgi/app/speedy_composer/reload
494 0 srw-rw-rw- 1 ubuntu ubuntu 0 Jul 20 08:41 /run/uwsgi/app/speedy_composer/socket
493 4 -rw-r--r-- 1 root root 5 Jul 20 08:41 /run/uwsgi/app/speedy_composer/pid
The new server:
$ find /run/uwsgi/ -ls
554 0 drwxr-xr-x 3 root root 60 Jul 20 10:15 /run/uwsgi/
555 0 drwxr-xr-x 6 root root 120 Jul 20 11:14 /run/uwsgi/app
573 0 drwxr-xr-x 2 ubuntu ubuntu 80 Jul 20 11:14 /run/uwsgi/app/speedy_net
598 0 srw-rw-rw- 1 ubuntu ubuntu 0 Jul 20 11:14 /run/uwsgi/app/speedy_net/socket
584 4 -rw-r--r-- 1 root root 5 Jul 20 11:14 /run/uwsgi/app/speedy_net/pid
568 0 drwxr-xr-x 2 ubuntu ubuntu 80 Jul 20 11:14 /run/uwsgi/app/speedy_match
572 0 srw-rw-rw- 1 ubuntu ubuntu 0 Jul 20 11:14 /run/uwsgi/app/speedy_match/socket
569 4 -rw-r--r-- 1 root root 5 Jul 20 11:14 /run/uwsgi/app/speedy_match/pid
565 0 drwxr-xr-x 2 ubuntu ubuntu 80 Jul 20 11:14 /run/uwsgi/app/speedy_mail
567 0 srw-rw-rw- 1 ubuntu ubuntu 0 Jul 20 11:14 /run/uwsgi/app/speedy_mail/socket
566 4 -rw-r--r-- 1 root root 5 Jul 20 11:14 /run/uwsgi/app/speedy_mail/pid
499 0 drwxr-xr-x 2 ubuntu ubuntu 80 Jul 20 11:14 /run/uwsgi/app/speedy_composer
564 0 srw-rw-rw- 1 ubuntu ubuntu 0 Jul 20 11:14 /run/uwsgi/app/speedy_composer/socket
561 4 -rw-r--r-- 1 root root 5 Jul 20 11:14 /run/uwsgi/app/speedy_composer/pid
The old server is Ubuntu 16.04.6 LTS, and the new server is Ubuntu 18.04.2 LTS.
I checked /var/log/uwsgi/app/speedy_net.log and it contains lots of lines. Here are the last 5 lines:
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c393) [0x7fe09b2ac393]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7fe09b277b70]
*** end of backtrace ***
Sun Aug 11 11:24:35 2019 - DAMN ! worker 2 (pid: 6874) died :( trying respawn ...
Sun Aug 11 11:24:35 2019 - Respawned uWSGI worker 2 (new pid: 6906)
The last 7 lines are now:
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyCFunction_FastCallDict+0x13a) [0x7f4d7c245e2a]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c393) [0x7f4d7c31a393]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
*** end of backtrace ***
Sun Aug 11 11:37:17 2019 - DAMN ! worker 2 (pid: 7915) died :( trying respawn ...
Sun Aug 11 11:37:17 2019 - Respawned uWSGI worker 2 (new pid: 7944)
Sun Aug 11 11:37:17 2019 - epoll_ctl(): File exists [core/event.c line 524]
And:
Sun Aug 11 11:42:56 2019 - epoll_ctl(): File exists [core/event.c line 524]
Sun Aug 11 11:42:56 2019 - epoll_ctl(): File exists [core/event.c line 524]
Sun Aug 11 11:42:56 2019 - !!! uWSGI process 8118 got Segmentation Fault !!!
Sun Aug 11 11:42:56 2019 - *** backtrace of 8118 ***
/usr/bin/uwsgi(uwsgi_backtrace+0x2a) [0x563d84f5098a]
/usr/bin/uwsgi(uwsgi_segfault+0x23) [0x563d84f50d73]
/lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7f4d7ff98f20]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(ssl3_cleanup_key_block+0xb) [0x7f4d773d4cab]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(ssl3_clear+0x16) [0x7f4d773d25a6]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/./libssl-c0c2ede4.so.1.0.2q(tls1_clear+0x9) [0x7f4d773de219]
/usr/lib/x86_64-linux-gnu/libssl.so.1.1(SSL_new+0x3f2) [0x7f4d80c11a22]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0x23969) [0x7f4d77669969]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0x24fc5) [0x7f4d7766afc5]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(PQconnectPoll+0xb78) [0x7f4d77654ba8]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(+0xfa28) [0x7f4d77655a28]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/.libs/libpq-bd31fe2b.so.5.11(PQconnectdb+0x1f) [0x7f4d7765841f]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so(+0x12b01) [0x7f4d778a9b01]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so(+0x1388f) [0x7f4d778aa88f]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x122342) [0x7f4d7c220342]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x89) [0x7f4d7c333e19]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_CallFunction_SizeT+0x124) [0x7f4d7c334ce4]
/home/ubuntu/speedy-net/env/lib/python3.6/site-packages/psycopg2/_psycopg.cpython-36m-x86_64-linux-gnu.so(+0xc258) [0x7f4d778a3258]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(PyCFunction_Call+0x96) [0x7f4d7c2460a6]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x7a50) [0x7f4d7c2e8450]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21bbab) [0x7f4d7c319bab]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x3e) [0x7f4d7c31a67e]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1beeb3) [0x7f4d7c2bceb3]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(PyObject_Call+0x48) [0x7f4d7c334b08]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x4001) [0x7f4d7c2e4a01]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c472) [0x7f4d7c31a472]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c472) [0x7f4d7c31a472]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c472) [0x7f4d7c31a472]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21bbab) [0x7f4d7c319bab]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c29e) [0x7f4d7c31a29e]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c472) [0x7f4d7c31a472]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21bbab) [0x7f4d7c319bab]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c29e) [0x7f4d7c31a29e]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x3ea5) [0x7f4d7c2e48a5]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1c85ec) [0x7f4d7c2c65ec]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1bc496) [0x7f4d7c2ba496]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1bc970) [0x7f4d7c2ba970]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x122342) [0x7f4d7c220342]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x89) [0x7f4d7c333e19]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c058) [0x7f4d7c31a058]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c472) [0x7f4d7c31a472]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21a953) [0x7f4d7c318953]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyFunction_FastCallDict+0x2c3) [0x7f4d7c319fb3]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x131) [0x7f4d7c333ec1]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_Call_Prepend+0xcd) [0x7f4d7c334a9d]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x89) [0x7f4d7c333e19]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x270821) [0x7f4d7c36e821]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0xb8ef5) [0x7f4d7c1b6ef5]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x1ec989) [0x7f4d7c2ea989]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyCFunction_FastCallDict+0x13a) [0x7f4d7c245e2a]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0x21c393) [0x7f4d7c31a393]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x5170) [0x7f4d7c2e5b70]
*** end of backtrace ***
Sun Aug 11 11:42:57 2019 - DAMN ! worker 2 (pid: 8118) died :( trying respawn ...
Sun Aug 11 11:42:57 2019 - Respawned uWSGI worker 2 (new pid: 8124)
Sun Aug 11 11:42:57 2019 - epoll_ctl(): File exists [core/event.c line 524]
The connection to the database (PostgreSQL) works from the command line, for example this code works fine:
import django
django.setup()
from django.contrib.sites.models import Site
for site in Site.objects.all().order_by('pk'):
print(site.id, site.domain)
Eventually it was https://stackoverflow.com/a/54253374/57952. I had to run the following 2 commands:
/home/ubuntu/speedy-net/env/bin/pip uninstall psycopg2
/home/ubuntu/speedy-net/env/bin/pip install --no-binary :all: psycopg2==2.7.7
And then the server started to work.