Search code examples
nginxflaskcherrypyspark-submit

error when runing deploying cherrypy with nginx


I created an application in flask and i deployed it with cherrypy

def run_server(app):

# Enable WSGI access logging via Paste
app_logged = TransLogger(app)
app.config['JSON_AS_ASCII'] = False 
# Mount the WSGI callable object (app) on the root directory
cherrypy.tree.graft(app_logged, '/')

# Set the configuration of the web server
cherrypy.config.update({
    'engine.autoreload.on': True,
    'log.screen': True,
    'server.socket_port': 5050,
    'server.socket_host': '0.0.0.0'
})

# Start the CherryPy WSGI web server
cherrypy.engine.start()
cherrypy.engine.block()

if __name__ == "__main__":
# Init spark context and load libraries
sc = init_spark_context()
app = create_app(sc)

# start web server
run_server(app)

I locked the port 5050 using nginx by adding this code

server {
    listen 5050;
    listen [::]:5050;
    server_name _;
    root /var/www/html;
    location / {
            try_files $uri $uri/ =404;
    }

}

to /etc/nginx/sites-available/default

I got this error when i run cherrypy ( or spark-submit)

port 5050 is not free

Solution

  • nginx & cherrypy (Python) are independent HTTP servers, so they each need their own port. You're seeing the port 5050 is not free error because those processes are competing for the same one. (In this case it looks like the nginx process started up first, so it "beat" cherrypy to the punch). Here's the relevant config:

    cherrypy

    cherrypy.config.update({
        'engine.autoreload.on': True,
        'log.screen': True,
        'server.socket_port': 5050,      # <-- port 5050
        'server.socket_host': '0.0.0.0'
    })
    

    nginx

    server {
        listen 5050;  # <-- port 5050
    

    Try something like this in your nginx config instead:

    server {
        listen 8080;
        server_name localhost;
    
        upstream my_cherrypy_svc {
            server localhost:5050;
        }
    
        location /foo {
            proxy_pass http://my_cherrypy_svc;
            proxy_redirect default;
        }
    }
    

    This creates an nginx server running on port 8080 that proxies /foo requests to the cherrypy server running on port 5050.

    In other words, for a hypothetical endpoint /bar implemented in your Flask app, you should see the same result for both of these calls:

    curl localhost:5050/bar      # hits the cherrypy server directly, bypassing nginx
    curl localhost:8080/foo/bar  # hits nginx, which matches on the `foo` prefix and forwards the request to cherrypy
    

    This is a little contrived but hopefully illustrates the main issue: nginx and cherrypy are independent servers, so they each need a port of their own. (This pattern is especially handy if you have multiple cherrypy servers running at once, because all internet traffic can be directed to nginx, which can then dispatch calls to the appropriate cherrypy server.)