Search code examples
pysparkhivenulldbnull

pyspark hive - Insert NULL as DB null through text file

While inserting text file from pyspark shell to hive table.

NULL values treating as string in table.

If i query hive table, records can be retried only with filter condition = 'NULL' rather than is null.

Can any one suggest how to insert data as DB NULLS in table

Solution

  • Check if your spark dataframe is having null or None. And while writing to the hive table set the nullValue option as 

    df.write.option('nullValue', None).saveAsTable(path)

    This will solve your issue.