Search code examples
pysparkdatabricksspark-structured-streamingdelta-lake

Trigger.AvailableNow for Delta source streaming queries in PySpark (Databricks)


All the examples in the Databricks documentation are in Scala. Can't find how to use this trigger type from PySpark. Is there an equivalent API or workaround ?


Solution

  • Python implementation missed the Spark 3.2 release, so it will be included into Spark 3.3 only (for OSS version). On Databricks it was released as part of DBR 10.3 (or 10.2?), and could be used as following:

    .trigger(availableNow=True)