Search code examples
pythondatepysparkdatediffsubtraction

Python Pyspark: Filter for 1 Day Before Current Date Using F.current_date()

I want to filter my data set for all the dates before a certain date. Specifically 1 day before the current date.

I tried the code below: 

df = df.filter(F.col('date') <= F.current_date() - 1)

But I got the following error: 

u"cannot resolve '(current_date() - 1)' due to data type mismatch: differing types in '(current_date() - 1)' (date and int)
Solution

  • F.date_sub method should work:

    df.filter(F.col('date') <= F.date_sub(F.current_date(), 1))