Search code examples
pythonpipdatabricks

How to force using a lower version python library in Databricks

I am trying to use huggingface transformers in Databricks runtime 13.3. However current version of transformers can't be imported because it conflicts with the version of urllib3 (2.1.0) that is preinstalled on the cluster. A lower version would work with transformers. However according to Databricks, none of the libraries that comes with the runtime can be uninstalled.

I tried something like this, but it doesn't let me switch. What is the correct way to force using lower version of a lib on Databricks

Solution

  • Use %pip install to update urllib3 version on your notebook:

    %pip install urllib3==1.26

    for more details see Notebook-scoped Python libraries