Search code examples
rapache-sparkspark-structured-streamingsparklyr

SparklyR Streaming and JDBC

Could you provide an example of using the SparklyR stream_read_jdbc():

stream_read_jdbc(sc, path, name = NULL, columns = NULL,
    options = list(), ...)

and stream_write_jdbc(), 

stream_write_jdbc(x, mode = c("append", "complete", "update"),
  trigger = stream_trigger_interval(),
  checkpoint = file.path("checkpoints", random_string("")),
  options = list(), ...)

preferably connecting to Oracle? If the sample uses any other database that is OK. Why is there a path parameter to a file while reading from JDBC? Is it not the point to source from a database instead of from the filesystem?

Solution

  • While spakrlyr indeed provide following functions:

    • stream_read_jdbc
    • stream_write_jdbc

    this is effectively a piece of dead code, as Spark Structured Streaming doesn't support streaming reads and writes with JDBC source.