Search code examples
pysparktimeazure-databricks

when i run a particular code i repeatedly getting the following error Azure Databricks

start_date_str = dbutils.widgets.get("startdate")
start_date = to_date(lit(start_date_str), 'yyyy-MM-dd')

enddate = (spark
    .range(1)
    .select(date_add(start_date, 365 * 10).alias("enddate"))
    .collect()[0]["enddate"]
)

(
  spark.sql(f"select explode(sequence(to_date('{start_date}'), to_date('{enddate}'), interval 1 day)) as calendarDate")
    .createOrReplaceTempView('dates')
)

HOW TO SOLVE THIS ERROR

ParseException: 
[PARSE_SYNTAX_ERROR] Syntax error at or near 'to_date'(line 1, pos 41)

== SQL ==
select explode(sequence(to_date('Column'), to_date('2009-12-29'), interval 1 day)) as calendarDate

(I want a correct code for this.)

Solution

  • Most probably this is caused by this piece of code: to_date('{start_date}').

    The problem is that start_date is the column object so when you print it inside the string you get it represented as Column<'to_date(2022-01-01, yyyy-MM-dd)'> that lead to the error.

    instead you can change to_date('{start_date}') to to_date('{start_date_str}', 'yyyy-MM-dd')