Situation: I would like to deploy our Django project's code changes to the stage or production server. For this we have an ansible deployment script, that pulls the django project's code from gitlab, migrates the database, collects the static files, restarts the different servers, etc.
Problem: Some uwsgi processes (1 out of 10 on average) do not reload my django application when I deploy via the ansible script. I can reproduce this for example after the static assets change after a code update and the static assets bundle's file name hashes changes. The server then throws a 500 error because uwsgi still tries to load the bundle with the old hash (via https://github.com/ezhome/django-webpack-loader). When I refresh the website in the browser after a deployment where the static assets have changed, then the server returns a 500 error in about 10 - 20% of the requests because it cannot find the old static assets bundle (example: bundle.9290aAFKASE234.js).
When I run /etc/init.d/supervisor restart on the server the problem immediately goes away, all uwsgi processes seem to be reloaded correctly.
relevant nginx config
location / {
{% if nginx_site_basic_auth %}
auth_basic "Test Server what.digital";
auth_basic_user_file "{{ project_root}}/nginx_passwdfile";
{% endif %}
uwsgi_pass {{ project_name }}_server;
uwsgi_read_timeout 1800s;
include /etc/nginx/uwsgi_params;
}
uwsgi config (app.ini)
# uwsgi.ini file
[uwsgi]
# Django-related settings
# the base directory (full path)
chdir = {{ project_root }}
# Django's wsgi file
module = {{ project_django_wsgi }}
plugins = python
# the virtualenv (full path)
home = {{ project_venv }}
# process-related settings
# prevents some pretty crazy erratic behaviour
lazy-apps = true
# master
master = true
# maximum number of worker processes
processes = 10
pidfile = {{ wsgi_pid_file }}
#touch-reload = {{ project_root }}/config/wsgi.py
#daemonize = {{ project_root }}/{{ project_name }}-uwsgi-daemon.log
logto = {{ project_root }}/{{ project_name }}-uwsgi-error.log
log-5xx = true
disable-logging = true
harakiri = 120
no-orphans = true
max-requests = 50
# give appropriate permissions to socket file
chmod-socket = 666
# the socket (use the full path to be safe)
socket = {{ project_root }}/{{ project_name }}.sock
# http-socket = :8000
# clear environment on exit
vacuum = true
buffer-size = 32768
# https://stackoverflow.com/questions/32452529/uwsgi-emperor-unicodeencodeerror-ascii-codec-cant-encode-character
env = LANG=en_US.UTF-8
supervisor template (populated by ansible):
[program:uwsgi]
user = {{ user_name }}
command={{ project_venv }}/bin/uwsgi --ini {{ project_root }}/{{ uwsgi_conf_file }}
environment={% for name, value in env_vars.iteritems() -%}
{{ name }}="{{ value|replace('%', '%%') }}";
{%- endfor %}
autostart=true
autorestart=true
stderr_logfile = {{ project_logs }}/supervisor_uwsgi_err.log
stdout_logfile = {{ project_logs }}/supervisor_uwsgi_stdout.log
stopsignal=INT
relevant ansible config
- name: upload supervisor configuration
template: src=supervisor.j2 dest=/etc/supervisor/conf.d/{{ project_name }}.conf
become_user: root
#- name: make sure supervisord runs
# service: name=supervisor state=started
# become_user: root
#- name: reread supervisorctl config files
# command: supervisorctl reread
# become_user: root
- name: restart supervisor
# ansible's supervisorctl module doesnt really restart and some weird caching problems emerge
# shell: supervisorctl restart uwsgi
shell: /etc/init.d/supervisor restart
become_user: root
- name: make sure nginx is restarted
service: name=nginx state=restarted
become_user: root
We finally found the reason for this: django is using local caching per default which is not safe to use when having multiple uwsgi processes. The solution is to set up a memcached service and configure django's CACHES for it as described here: https://docs.djangoproject.com/en/4.0/topics/cache/#memcached