How can I connect mlflow server via nginx ssl authentication?

Question

System information OS Platform and Distribution: Windows 10 MLflow installed: using pip MLflow version: version 1.24.0 **Python version: Python 3.9.7 **

Describe the problem I have created a docker-compose system with a backend/artifact storages, mlflow server and nginx to add an authentication layer.

...
mlflow:
        restart: always
        build: .
        environment:
            - AWS_ACCESS_KEY_ID=${MINIO_USR}
            - AWS_SECRET_ACCESS_KEY=${MINIO_PASS}       
        expose:
            - '5000'
        networks:
            - frontend
            - backend
        depends_on:
            - storage                       
        image: 'mlflow:Dockerfile'
        container_name: mlflow_server_nginx

    nginx:
        restart: always
        build: ./nginx
        container_name: mlflow_nginx
        ports:
            - 5043:443
        links:
            - mlflow:mlflow
        volumes:
            - 'path/to/nginx/auth:/etc/nginx/conf.d'
            - 'path/to/nginx/nginx.conf:/etc/nginx/nginx.conf:ro'
        networks:
            - frontend
        depends_on:
            - mlflow

I have created an user/password via htpasswd and a custom SSL CA (.pem/.key) using openssl and my-mlflow.com server-name.

When the docker-compose system is built i can access to mlflow UI via my browser. But when i try to create a new experiment using python trying diferent approaches, i get next errors: Executed code 1:

# Setting the requried environment variables
os.environ['MLFLOW_S3_ENDPOINT_URL'] = 'https://localhost:9000'
os.environ['AWS_ACCESS_KEY_ID'] = 'user'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'password'
# Set username and password for added authentication
#os.environ['MLFLOW_TRACKING_URI '] = 'https://localhost:5043/'
#os.environ['MLFLOW_TRACKING_USERNAME '] = 'user'
#os.environ['MLFLOW_TRACKING_PASSWORD '] = 'password'
#os.environ['MLFLOW_TRACKING_SERVER_CERT_PATH'] = 'path/to/nginx/auth/domain.pem'
#os.environ['MLFLOW_TRACKING_CLIENT_CERT_PATH'] = 'path/to/nginx/auth/domain.pem'
# MLflow enviroment
remote_server_uri = "https://user:password@localhost:5043/" # set to your server URI
mlflow.set_tracking_uri(remote_server_uri)

mlflow.set_experiment("MLflow_demo")

Error:

MlflowException: API request to https://user:password@localhost:5043/api/2.0/mlflow/experiments/list failed with exception HTTPSConnectionPool(host='localhost', port=5043): Max retries exceeded with url: /api/2.0/mlflow/experiments/list?view_type=ALL (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1108)')))

After read some notes in the documentation and realated issues I tryed next

# Setting the requried environment variables
os.environ['MLFLOW_S3_ENDPOINT_URL'] = 'https://localhost:9000'
os.environ['AWS_ACCESS_KEY_ID'] = 'user'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'password'
# Set username and password for added authentication
#os.environ['MLFLOW_TRACKING_URI '] = 'https://localhost:5043/'
#os.environ['MLFLOW_TRACKING_USERNAME '] = 'user'
#os.environ['MLFLOW_TRACKING_PASSWORD '] = 'password'
#os.environ['MLFLOW_TRACKING_SERVER_CERT_PATH'] = 'path/to/nginx/auth/domain.pem'
os.environ['MLFLOW_TRACKING_CLIENT_CERT_PATH'] = 'path/to/nginx/auth/domain.pem'
# MLflow enviroment
remote_server_uri = "https://user:password@localhost:5043/" # set to your server URI
mlflow.set_tracking_uri(remote_server_uri)

mlflow.set_experiment("MLflow_demo")

Error:

MlflowException: API request to https://user:password@localhost:5043/api/2.0/mlflow/experiments/list failed with exception HTTPSConnectionPool(host='localhost', port=5043): Max retries exceeded with url: /api/2.0/mlflow/experiments/list?view_type=ALL (Caused by SSLError(SSLError(9, '[SSL] PEM lib (_ssl.c:4012)')))

Finally

# Setting the requried environment variables
os.environ['MLFLOW_S3_ENDPOINT_URL'] = 'https://localhost:9000'
os.environ['AWS_ACCESS_KEY_ID'] = 'user'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'password'
# Set username and password for added authentication
#os.environ['MLFLOW_TRACKING_URI '] = 'https://localhost:5043/'
#os.environ['MLFLOW_TRACKING_USERNAME '] = 'user'
#os.environ['MLFLOW_TRACKING_PASSWORD '] = 'password'
os.environ['MLFLOW_TRACKING_SERVER_CERT_PATH'] = 'path/to/nginx/auth/domain.pem'
#os.environ['MLFLOW_TRACKING_CLIENT_CERT_PATH'] = 'path/to/nginx/auth/domain.pem'
# MLflow enviroment
remote_server_uri = "https://user:password@localhost:5043/" # set to your server URI
mlflow.set_tracking_uri(remote_server_uri)

mlflow.set_experiment("MLflow_demo")

Error:

MlflowException: API request to https://user:password@localhost:5043/api/2.0/mlflow/experiments/list failed with exception HTTPSConnectionPool(host='localhost', port=5043): Max retries exceeded with url: /api/2.0/mlflow/experiments/list?view_type=ALL (Caused by SSLError(SSLCertVerificationError("hostname 'localhost' doesn't match '*.my-mlflow.com'")))

Can you give me some hints about how to solve it?

Thank you very much! Fernando....

Answer 1

You can set:

os.environ['MLFLOW_TRACKING_INSECURE_TLS'] = 'true'

And then try to get your cert-chain straight from there for production use.

Also see Documentation: https://mlflow.org/docs/latest/tracking.html#id19