Rename written CSV file Spark throws Error "Path must be absolute" - Azure Data Lake

Question

I tried solution described in Rename written CSV file Spark but I am getting the following error "java.lang.IllegalArgumentException: Path must be absolute". How could I fix it? It can be in scala or Python code. Thanks :)

import org.apache.hadoop.fs._
val fs = FileSystem.get(sc.hadoopConfiguration)

var table_name = dbutils.widgets.get("table_name")

val filePath = "mnt/datalake/" + table_name + "/"

print("file path: " + filePath)

val fileName = fs.globStatus(new Path(filePath+"part*"))(0).getPath.getName
print("file name: " + fileName)

fs.rename(new Path(filePath+fileName), new Path(filePath+"file.csv"))

Outputs:

file path: mnt/datalake/MyTable/
file name: part-00000-tid-9118XXX-c000.csv

Error

java.lang.IllegalArgumentException: Path must be absolute: mnt/datalake/MyTable/part-00000-tid-9118XXXXc000.csv

Answer 1

try this:

import org.apache.hadoop.fs._
import org.apache.hadoop.fs.{FileSystem, Path}
val fs = FileSystem.get(sc.hadoopConfiguration)
val filePath = "dbfs:/FileStore/tables/part_00000-6a99e/"
val fileName = fs.globStatus(new Path(filePath))(0).getPath.getName
fs.rename(new Path(filePath+fileName), new Path(filePath+"file.csv"))