Search code examples
sqlapache-sparkpysparkapache-drill

Apache spark jdbc connect to apache drill error

I am sending query to apache drill from apache spark. I am getting the following error:

java.sql.SQLException: Failed to create prepared statement: PARSE ERROR: Encountered "\"" at line 1, column 23.

When traced, I found I need to write a custom sql dialect. The problem I do not find any examples for pyspark. All the examples are for scala or java. Any help is highly appreciated.!

Here is the pyspark code :

`dataframe_mysql = spark.read.format("jdbc").option("url", "jdbc:drill:zk=ip:2181;schema=dfs").option("driver","org.apache.drill.jdbc.Driver").option("dbtable","dfs.`/user/titanic_data/test.csv`").load()`
Solution

  • I navigated to the drill web ui and updated the planner.parser.quoting_identifiers parameter to ". Then I edited my query as below:

    dataframe_mysql = spark.read.format("jdbc").option("url", "jdbc:drill:zk=ip:2181;schema=dfs;").option("driver","org.apache.drill.jdbc.Driver").option("dbtable","dfs.\"/user/titanic_data/test.csv\"").load()

    And it worked like charm!