spark.read.load from postgres database how to limit(10)

Question

df = spark.read.format("jdbc") \
                      .option("url", url) \
                      .option("dbtable", "purchases") \
                      .load() \
                      .limit(10)

I'm executing the above code, and it seems to me that it's trying to load the entire table before applying the limit.

Where should my limit be placed in order to do this correctly?

Answer 1

I figured it out. I put the query option in with my SQL.

df = spark.read.format("jdbc") \
                          .option("url", url) \
                          .option("query", "select * from po_poterm limit 100") \
                          .load()