Search code examples
pythonapache-spark-sqlazure-databricksdelta-lake

Spark sql throws error while reading CDF enabled DELTA table in Azure databricks


I am trying to run the below query in python notebook inside azure databricks

tab ='db.t1'

df =spark.sql(f"SELECT MAX(_commit_version) as max_version FROM table_changes({tab},0)")

df.first()["max_version"]

But it throws error as below

AnalysisException: [UNRESOLVED_COLUMN.WITHOUT_SUGGESTION] A column or function parameter with name `t1` cannot be resolved. ; line 1 pos 62;
'Project ['MAX('_commit_version) AS max_version#5374]
+- 'UnresolvedTableValuedFunction [table_changes], ['db.t1, 0]

Can some some one help me


Solution

  • Table name parameter should be in the quotes (see docs). Try:

    df =spark.sql(f"SELECT MAX(_commit_version) as max_version FROM table_changes('{tab}',0)")