Search code examples
rethinkdbrethinkdb-python

RethinkDB Changefeeds Lost Connection

I registered a Changefeeds to one of my RethinkDB tables, and it's been working perfectly for the past 6 days. Data get posted to my rethinkDB table every 2 minutes, and the Changefeeds has no problem getting them. But yesterday due to some reason the data uploading stopped for about an hour. During that period, Changefeeds didn't get any changes, which is expected. The problem is that after the data uploading restarted, I still cannot get any changes from rethinkDB. It is lost forever. The problem is gone when I restarted my program. The code is like:

def get_rdb_connection():
    "Helper function for get rdb connection"
    try:
        rdb_conn = rdb.connect(host='rethinkdb_ip', port=28015, db='test')
        logger.info('connection established.')
        return rdb_conn
    except RqlDriverError:
        logger.err("No rdb database connection could be established.")
        return

conn = get_rdb_connection()
feed = rdb.table('skydata').changes(squash=False).run(conn)
for change in feed:
    logger.info("change detected")
    """ do some stuff """

The output:

change detected
change detected
change detected
     ...
change detected

Then the output just stopped.

I'm pretty sure the code in """ do some stuff """ doesn't cause any block, since it is pretty simple and has been running for weeks. And Supervisord status also shows it is always in running state.

So I wonder if there's timeout-like mechanism, if there's no change for a certain period, it will stop listening?

Edit:

I think it may be due to connection loss since there's no change for more than an hour and the program is just doing nothing. I'm not sure how rethinkdb manages the connection to its database. Is the connection kept alive automatically? Will it be reestablished if it gets lost?

Solution

  • The connection shouldn't be lost in that scenario, or if it is you should receive an error in your driver rather than the changefeed silently dropping changes.

    On the face of it this sounds like a bug; I opened https://github.com/rethinkdb/rethinkdb/issues/4572 to track it. If it isn't too much trouble, could you add some more details on your setup to that issue? (What OS you're using, what your network setup is, whether this bug is regularly reproducible for you, etc.)