Pyspark: Data frame has a salary column with float type and some NaN values, while filtering out salaries NaN values also coming in the final output

Question

Using the filter method on the salary column, I got the NaN values; as far as I know, NaN values should not be coming in the output data frame. I also asked from ChatGPT, and it shows me without NaN values. When asking for the output difference, it says that could be due to a version mismatch, but that is not the case. BTW, I'm using Spark v3.3.2

users = {
    'name': ['John', 'Jane', 'Mike'],
    'salary': [400.0, None, 200.0]
}

pdf = pd.DataFrame(users)
sdf = spark.createDataFrame(pdf)


# filter out the rows with salaries greater than 300
sdf_filtered = sdf.filter(sdf.salary > 300)
sdf_filtered.show()

My Output

+----+------+
|name|salary|
+----+------+
|John| 400.0|
|Jane|   NaN|
+----+------+

ChatGPT shows me (with version v3.2)

+------+
|salary|
+------+
| 400.0|
+------+

I think ChatGPT does not produce the right output here, or my output is wrong.

Answer 1

Because Nan by construction belongs to the class float, also it represents a missing value, so when filtering on salary > 300, spark will found that the NaN value has a numerical type (float), and also is missing, so he's not sure if it's bigger than 300 or not (undefined value), so he will not make the decision for you and will keep that value, if you want to remove NaN's you can filter them out with this your query:

sdf_filtered = sdf.filter(~isnan(sdf.salary) & (sdf.salary > 300))

For the ChatGPT part, I will not even consider its answer, because it has been shown that it can give wrong results.