Search code examples
pysparkapache-spark-sql

pyspark query on case statement throws error

I am running the below pyspark query

It throws the below error . I want dynamic values to passes via a varaibale in to case statement

 data1 = 'HI'
 data2 = 'HELLO'

 spark.sql(f""" select  Initials,
                case when Initials  in ('AM','NJ') then {data1} else {data2} end as s  
                from omegawriter""").display()

I get below error

  A column or function parameter with name `HI` cannot be resolved.
  Project [Initials#69, CASE WHEN Initials#69 IN (AM,NJ) THEN 'HI ELSE 'HELLO END AS s#57]

I dont understand what is the problem with this query . Can someone help me please

Solution

  • Plain text values need to be quoted.

     spark.sql(f""" select  Initials,
                case when Initials  in ('AM','NJ') then '{data1}' else '{data2}' end as s  
                from omegawriter""").display()