Search code examples
pythonazure-cognitive-services

CSV ingestion error in Azure Data Bricks - File read error

My task is to perform data ingestion of CSV file using python. I am using the following code block and getting error of file not found eventhough the location is right.

spark.read.format("Filelocation") \
  .option("file.format", "csv") \
  .option("file.schemaLocation", "checkpoint") \
  .load("source_data") \
  .writeStream \
  .option("mergeSchema", "true") \
  .option("My_Checkpoint", "Checkpoint path") \
  .start("target")

I have no clue of the error. Any help is appreciated.

Solution

  • It's a simple read statement error that I have observed from your code block.

    spark.readStream.format("Filelocation")

    Use the above code block in the first line instead of spark.read.format("Filelocation") and remaining statements are as it is.