I'm writing an application that integrates with Zookeeper for client-side service discovery. The application is a Python Flask app deployed with gunicorn and Docker in a K8s cluster. It uses the kazoo Zookeeper client. Because of the layers (gunicorn, Docker, and K8s), I'm unable to get a kazoo watch function to work to automatically update the service endpoint when there a change. And, I don't want to poll Zookeeper every time I use the service because that would be inefficient.
I know that the change in service endpoint happens only a few times a day. So, I'm considering this approach: every time I call the service (using a request GET), if the response shows unreachable URL (4xx error), I'll contact Zookeeper for a new service endpoint and try again. My question is, has anyone used this approach in practice?
Are there any other client-side service discovery approaches I should be considering when integrating with Zookeeper?
Upon further research and experimentation, I found that the alternative approach wasn't necessary. I got the watch function to work.
The watch function wasn't working for me originally because I was using the preload option in gunicorn and creating the KazooClient object before worker processes were forked. This meant that the gunicorn workers were trying to share the KazooClient object, which led to watch functions not working.
The fix was to delay the creation of the KazooClient object by doing this creation in the "before_first_request" function of the Flask app, which makes the creation to occur after the gunicorn worker processes are forked.
(I know the "before_first_request" function is being deprecated in Flask, but for now it works)