Search code examples
djangogunicorngevent

django + gunicorn: problem with async workers (gevent)


I am having the exact same problem as this post Django+gunicorn+nginx upload large file 502 error. But the solution provided does not make the trick for me, maybe because it is fairly old.

I am using django, gunicorn, supervisor and nginx with python 3.6, I have installed gevent according to gunicorn's documentation, and my gunicorn config looks like that:

[program:gunicorn]
directory=/home/ubuntu/mysite
command=/home/ubuntu/env/bin/gunicorn --workers 3 --worker-class gevent --worker-connections=1000  --timeout 600 --bind unix:/home/ubuntu/mysite/app.sock app.wsgi:application

autostart=true
autorestart=true
stderr_logfile=/var/log/gunicorn/gunicorn.err.log
stdout_logfile=/var/log/gunicorn/gunicorn.out.log


[group:guni]
programs:gunicorn

and in gunicorn log, I get the error:

 File "/home/ubuntu/exo/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 16, in <module>
    raise RuntimeError("gevent worker requires gevent 1.4 or higher")
RuntimeError: gevent worker requires gevent 1.4 or higher

[2020-05-16 13:22:40 +0000] [24451] [DEBUG] Current configuration:
  config: None
  bind: ['unix:/home/ubuntu/myapp/app.sock']
  backlog: 2048
  workers: 4
  worker_class: sync
  threads: 12
  worker_connections: 1000
  max_requests: 0
  max_requests_jitter: 0
  timeout: 600
  graceful_timeout: 90
  keepalive: 2
  limit_request_line: 4094
  limit_request_fields: 100
  limit_request_field_size: 8190
  reload: False
  reload_engine: auto
  reload_extra_files: []
  spew: False
  check_config: False
  preload_app: False
  sendfile: None
  reuse_port: False
  chdir: /home/ubuntu/myapp
  daemon: False
  raw_env: []
  pidfile: None
  worker_tmp_dir: None
  user: 0
  group: 0
  umask: 0
  initgroups: False
  tmp_upload_dir: None
  secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
forwarded_allow_ips: ['127.0.0.1']
  accesslog: None
  disable_redirect_access_to_syslog: False
  access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
  errorlog: -
  loglevel: DEBUG
  capture_output: False
  logger_class: gunicorn.glogging.Logger
  logconfig: None
  logconfig_dict: {}
  syslog_addr: udp://localhost:514
  syslog: False
  syslog_prefix: None
  syslog_facility: user
  enable_stdio_inheritance: False
  statsd_host: None
  dogstatsd_tags:
  statsd_prefix:
  proc_name: None
default_proc_name: myapp.wsgi:application
  pythonpath: None
  paste: None
  on_starting: <function OnStarting.on_starting at 0x7f7836ae4bf8>
  on_reload: <function OnReload.on_reload at 0x7f7836ae4d08>
  when_ready: <function WhenReady.when_ready at 0x7f7836ae4e18>
  pre_fork: <function Prefork.pre_fork at 0x7f7836ae4f28>
  post_fork: <function Postfork.post_fork at 0x7f7836b000d0>
  post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f7836b001e0>
  worker_int: <function WorkerInt.worker_int at 0x7f7836b002f0>
  worker_abort: <function WorkerAbort.worker_abort at 0x7f7836b00400>
  pre_exec: <function PreExec.pre_exec at 0x7f7836b00510>
  pre_request: <function PreRequest.pre_request at 0x7f7836b00620>
  post_request: <function PostRequest.post_request at 0x7f7836b006a8>
  child_exit: <function ChildExit.child_exit at 0x7f7836b007b8>
  worker_exit: <function WorkerExit.worker_exit at 0x7f7836b008c8>
  nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f7836b009d8>
  on_exit: <function OnExit.on_exit at 0x7f7836b00ae8>
  proxy_protocol: False

Being a novice with this stuff it's really difficult to find out what's going on.

I have installed gevent see below:

Installing collected packages: greenlet, psutil, gevent
Successfully installed gevent-20.5.0 greenlet-0.4.15 psutil-5.7.0

UPDATE: I have modified gunicorn config.py file in /vev/lib/python3.6/site-packages/gunicorn/config.py

BASE_DIR = "/path/to/base/dir/"
sys.path.append(BASE_DIR)


bind = '127.0.0.1:8000'
backlog = 2048


import multiprocessing
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = 'gevent'
worker_connections = 1000
timeout = 300
keepalive = 2

and then modified my supervisor/conf/gunicorn.conf file as such:

[program:gunicorn]

directory=/home/ubuntu/mysite
command=/home/ubuntu/exo/bin/gunicorn  --config /home/ubuntu/venv/lib/python3.6/site-packages/gunicorn/config.py  unix:/home/ubuntu/mysite/app.sock mysite.wsgi:application

autostart=true
autorestart=true
stderr_logfile=/var/log/gunicorn/gunicorn.err.log
stdout_logfile=/var/log/gunicorn/gunicorn.out.log


[group:guni]
programs:gunicorn

This still gives me the same error. I am not even sure that if gunicorn config file was the right file to add the modifications but at this point I am out of things to try, maybe a fresh eye could unblock situation, or I am also open for any substitute that someone may know of


Solution

  • Finally I found out, hopefully this answer can become useful to someone who stumble upon this post with the same problem.

    1) need to install gevent as follow:

    python3 -m pip install gevent 
    

    2) in your env/python/site-packages/gunicorn/ add the following to config.py like:

    BASE_DIR = "/path/to/base/dir/"
    sys.path.append(BASE_DIR)
    
    
    bind = '127.0.0.1:8000'
    backlog = 2048
    
    
    import multiprocessing
    workers = multiprocessing.cpu_count() * 2 + 1
    worker_class = 'gevent'
    worker_connections = 1000
    timeout = 300
    keepalive = 2
    

    3) then in gunicorn.conf, gotta make sure to reference to the path of the config.py file

    [program:gunicorn]
    
    directory=/home/ubuntu/mysite
    command=/home/ubuntu/exo/bin/gunicorn  --config /home/ubuntu/venv/lib/python3.6/site-packages/gunicorn/config.py  --bind unix:/home/ubuntu/mysite/app.sock mysite.wsgi:application
    
    autostart=true
    autorestart=true
    stderr_logfile=/var/log/gunicorn/gunicorn.err.log
    stdout_logfile=/var/log/gunicorn/gunicorn.out.log
    
    
    [group:guni]
    programs:gunicorn