add column to existing dataframe from widgets values using pyspark

Question

I have a dataframe where i need to add a column from the widget value that is being passed. I am trying the below code but its not helping in anyways. When we display(pdf) we should also see the ID column has also been added.

  pdf=pTF.withColumn('ID',lit(dbutils.widgets.text("ID", "eiheifhj", label="pro_ID")).cast(StringType()))

The above code is puhsing all nulls into the columns ID. Could you please help. I would request the experts to help in the above challenge

Answer 1

In the following code, you are actually defining the widget instead of fetching the value the widget contains.

dbutils.widgets.text("ID", "eiheifhj", label="pro_ID")

• "ID" is the name of your widget.
• "eiheifhj" is the default value of the widget.
• "pro_ID" is the widget label in UI

so if you want to fetch the value of the 'ID' widget, you can use the following method.

dbutils.widgets.get(<widget_name>)

hence your program should be modified as follows :

pdf=pTF.withColumn('ID',lit(dbutils.widgets.get("ID")).cast(StringType()))