Search code examples
pythonairflowamazon-sagemakerurllib3

Broken DAG: urllib3 1.25.3 (/home/ubuntu/.local/lib/python3.7/site-packages), Requirement.parse('urllib3<1.25,>=1.21'), {'sagemaker'}

I have created a DAG in Airflow with SageMakerOperators and I have not been able to make them work. The title is the error that appears in the airflow GUI. For solving it, I have made the following tries:

sudo pip3 uninstall urllib3 && sudo pip3 install urllib3==1.22 
sudo pip3 install urllib3==1.22 --upgrade
sudo pip3 install urllib3==1.22 -t /home/ubuntu/.local/lib/python3.7/site-packages -upgrade

But I am still getting the error in the GUI. Plus, in the console of the webserver I am getting:

FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.local/lib/python3.7/site-packages/urllib3-1.22.dist-info/METADATA'

The thing is that if I make pip3 show urllib3 I get the version 1.22: enter image description here

However, it says dist-packages instead of site-packages. In addition, trying to go to /home/ubuntu/.local/lib/python3.7/site-packages/urllib3-1.22.dist-info/ for trying to solve the metadata file not found error, the directory does not exists. enter image description here enter image description here

I am totally lost at this point. How could I solve this problem?

Solution

  • Here you go.

    Airflow is looking in the local (user) Python installation for the library but urllib3 is installed for all users. It's weird but try doing pip3 install --user urllib3==1.22.