Search code examples
dataframepysparkapache-spark-sql

How to use LIMIT ALL with DataFrame

When using Spark SQL I can use LIMIT ALL to return all rows. Is there an equivalent when using the DataFrame API so that I can do something like df.limit("ALL")?

Solution

  • According to the documentation, you have to pass an integer to the num parameter for df.limit(num)

    I am not sure about your use case, but if you want to parametrize num before calling df.limit(num), you could do something like the following:

    if isinstance(num, int):
    return df.limit(num)
elif num == 'ALL':
    return df
else:
    raise TypeError("Invalid type for num")