Search code examples
apache-sparkamazon-s3delta-lakeminio

DeltaTable.create fails with Minio identifier due to syntax error


Take this code: Java code

Here's the output:

boolean is false
Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: 
Syntax error at or near ':'(line 1, pos 3)

== SQL ==
s3a://spark-test/delta-table-three
---^^^

Why is the identifier fine when checking if it exists, but gives syntax error when using to create a table? BTW, if I check if a known table exists, it returns true as expected. So the isDeltaTable works just fine with the identifier.

*Note: I'm using Minio as my storage and not AWS.


Solution

  • when you use tableName it should be registered table name, like, in Hive Metastore (doc). In your case you provide a path to a bucket, so you need to use the location function instead (doc), as it's shown in the docs:

    DeltaTable.createOrReplace(spark) \
      .addColumn("...", "INT") \
      .location("s3a://bucket/path") \
      .execute()