Search code examples
databricksazure-databricksdatabricks-connect

databricks-connect failed to connect to Databricks cluster runtime 8.4

I have the following setup

Databricks Cluster:

8.4 ML (includes Apache Spark 3.1.2, Scala 2.12)

Client side:

databricks==0.2
databricks-cli==0.14.3
databricks-connect==8.1.10 (The latest version up to date 20210813)

When I run databricks-connect test to test the connection, I got this error message

java.lang.IllegalArgumentException: 
The cluster is running server version `dbr-8.4` but this client only supports Set(dbr-8.1). 
You can find a list of client releases at https://pypi.org/project/databricks-connect/#history, and install the right client version with `pip install -U databricks-connect==<version>`. 
For example, to install the latest 5.1 release, use `pip install -U databricks-connect==5.1.*`. 
To ignore this error and continue, set DEBUG_IGNORE_VERSION_MISMATCH=1.

According to the error message, I understand there is a compatibility issue with the client and server versions. However, I am already using the latest client.

  1. Does it mean the client doesn't add support for server version 8.4?
  2. Is there any way to circumvent this issue?
Solution

  • Unfortunately, it's the latest version of databricks-connect for DBR 8.x., and versions for 8.2-8.4 aren't released. But it may work if you add DEBUG_IGNORE_VERSION_MISMATCH=1 before the databricks-connect test or other commands like, pyspark, spark-submit, etc.