Flask socketio debug with eventlet and Redis spawns extra greenthreads?

Question

I'm trying to put together a simple Flask / socketio / eventlet server that subscribes to Redis events. The behavior I'm seeing is that with Flask debug enabled, every time Werkzeug detects changes and restarts socketio, another one of my redis listeners is started as well (except the old listener doesn't exit).

Here's a working version with all of the socketio handlers removed:

import json
from flask import Flask, render_template
from flask_socketio import SocketIO, emit
from flask.ext.redis import FlaskRedis
import eventlet
eventlet.monkey_patch()

with open('config/flask.json') as f:
    config_flask = json.load(f)

app = Flask(__name__, static_folder='public', static_url_path='')
app.config.update(
      DEBUG= True,
      PROPAGATE_EXCEPTIONS= True,
      REDIS_URL= "redis://localhost:6379/0"
    )

redis_cache = FlaskRedis(app)
socketio = SocketIO(app)

@app.route('/')
def index():
    cache = {}
    return render_template('index.html', **cache)

def redisReader():
    print 'Starting Redis subscriber'
    pubsub = redis_cache.pubsub()
    pubsub.subscribe('msg')
    for msg in pubsub.listen():
        print '>>>>>', msg

def runSocket():
    print "Starting webserver"
    socketio.run(app, host='0.0.0.0')

if __name__ == '__main__':
    pool = eventlet.GreenPool()
    pool.spawn(redisReader)
    pool.spawn(runSocket)
    pool.waitall()

Throw in some manual redis-cli publishing (PUBLISH msg himom) This produces the following output:

Starting Redis subscriber
Starting webserver
 * Restarting with stat
>>>>> {'pattern': None, 'type': 'subscribe', 'channel': 'msg', 'data': 1L}
Starting Redis subscriber
Starting webserver
 * Debugger is active!
 * Debugger pin code: 789-323-740
(22252) wsgi starting up on http://0.0.0.0:5000
>>>>> {'pattern': None, 'type': 'subscribe', 'channel': 'msg', 'data': 1L}
>>>>> {'pattern': None, 'type': 'message', 'channel': 'msg', 'data': 'himom'}
>>>>> {'pattern': None, 'type': 'message', 'channel': 'msg', 'data': 'himom'}

Why is the Redis listener getting started multiple times? If I make changes and save them, Werkzeug will start another one every time. How do I deal with this correctly?

Here's a list of the packages involved and their versions:

• Python 2.7.6
• Flask 0.10.1
• Werkzeug 0.11.4
• eventlet 0.18.4
• greenlet 0.4.9
• Flask-Redis 0.1.0
• Flask-SocketIO 2.2

** UPDATE ** I now have a partial solution. Everything above stays the same, except the pool behavior has been moved into Flask's 'before_first_request' function:

def setupRedis():
    print "setting up redis"
    pool = eventlet.GreenPool()
    pool.spawn(redisReader)

def runSocket():
    print "Starting Webserver"
    socketio.run(app, host='0.0.0.0')

if __name__ == '__main__':
    app.before_first_request(setupRedis)
    print app.before_first_request_funcs
    runSocket()

The remaining issue is that 'before_first_request' does not handle the case where there are existing websockets, but that is a separate question.

Answer 1

The solution here was to put my threading into a function called by Flask:

def setupRedis():
    pool = eventlet.GreenPool()
    pool.spawn(redisReader)
...
app.before_first_request(setupRedis)

This solved the extra threads left behind on Werkzeug restarts.