After removing a library jar added with the choice "Install automatically on all clusters" the library is still auto installed on all clusters in the workspace. I need to remove or disable the auto install behaviour of the library.
I imported a library from Maven to my Azure Databricks workspace. Once a new version came out I added the new version and just deleted the jar of the old version from the workspace.
The old version was added to the workspace with the "Install automatically on all clusters" behaviour and still gets installed on all new clusters created in the workspace even though the jar is gone from the GUI. Since the jar is gone (from the GUI) I can't change the "Install automatically on all clusters" behaviour of the jar and Databricks does not allow you to manually deattach auto installed libraries from a cluster (from the cluster view).
I tried readding the same library (same version) from Maven central and install the new jar on a cluster to see if that overrode the old installation hoping I could change the behaviour that way. Unfortunately when I try to add the newly downloaded version to a cluster with the library already auto installed the installation gets stuck at pending indefinately.
Any ideas on how to remedy this without having to setup a new workspace would be great.
This is a bug in Databricks (and it looks like it has been around for long time!). If you have ticked Auto install on all Clusters and then delete the shared libraries it installs it on all new clusters and cannot be uninstalled.
To uninstall it, go to the Shared Workspace and Create a new Library. Enter the exact same package and click on Create. On the next screen you can now untick "Install automatically on all clusters" and remove it from clusters where it is already installed. You can then move the new library to the trash.
Kudos to this post on the Databricks forums for finding this out: https://forums.databricks.com/questions/16400/remove-auto-install-library-thats-already-been-del.html