Loosing connection with Redis(aioredis) inside aiohttp app

Question

I am building a text/event-stream based view in aiohttp and also using pub-sub from Redis in aioredisimplementaion. It looks something like:

Script that get some data from server and publish into chanell

def main(host, port):
    server_logger.info('Got params connection host {0}, port {1}'.format(host, port))
    loop = asyncio.get_event_loop()
    title = None
    redis = loop.run_until_complete(create_redis(('localhost', 6379)))
    while True:
        new_title = loop.run_until_complete(get_title(host, port))
        if new_title != title:
            loop.run_until_complete(redis.publish('CHANNEL', new_title))
            title = new_title
    loop.close()
    return False

aiohttp view that subscribe for channel and write it into Stream response

stream = web.StreamResponse()
stream.headers['Content-Type'] = 'text/event-stream'
stream.headers['Cache-Control'] = 'no-cache'
stream.headers['Connection'] = 'keep-alive'

await stream.prepare(request)

redis = await create_redis(('localhost', 6379))
channel = (await redis.subscribe('CHANNEL'))[0]

while await channel.wait_message():
        message = await channel.get()
        if message:
            stream.write(b'event: track_update\r\n')
            stream.write(b'data: ' + message + b'\r\n\r\n')
        else:
            continue

And I got a lot of times something like :

DEBUG:aioredis:Creating tcp connection to ('localhost', 6379)

So sonnection is lost which also cause concurrent.futures.CancelledError and keep-alive connection will be lost. Is that ok that connection is lost such often? I was expecting to have persistent connection, sorry if I am missing something.

Answer 1

At first creation new redis connection in request handler is bad idea. Please use a connection pool per application.

You might get https://github.com/KeepSafe/aiohttp/blob/master/demos/polls/aiohttpdemo_polls/main.py as a sketch for recommended design principles.

Regarding to keep-alived connections -- they are not very persistent but closed after 75 secs inactivity period by default.

You may increase the period by passing keep_alive=300 parameter into app.make_handler() call but setting very big value is not robust -- during TCP nature connection may be broken without notifications under some conditions. Better to keep reasonable slow timeout and send custom ping requests to the server periodically if you have no data to pass.