Synapse Spark write to different /mount point [or] container

Am trying to write DataFrame from Synapse spark as .delta into a container.

Here is the example code:

sqlstmt = 'select * from tableblah limit 5'
df = spark.sql(sqlstmt)
df.write.format("delta").mode("overwrite").save("/bronze/account")

What this is doing is it's into Synapse's Primay ADLS Gen2 file system (ex: container name 'synapsews', now file written is in synapsews container /bronze/account.

How can i mount different container and write into it.

I tried mounting 'devbronze' container as below:

mssparkutils.fs.mount(
"abfss://devbronze@dlsdataxxxxdirdev001.dfs.core.windows.net",
"/devbronze",
{ "linkedService":"LS_synapsews" }

)

now when tried to write this mount as below:

df.write.format("delta").mode("overwrite").save("/devbronze/account")

it still write to Primay ADLS Gen2 file system which is synapsews by creating /devbronze/account folders.

How to change to write to mounted point 'devbronze'

Thanks

Solution

I tried the same way as yours in my environment and ended up with same result. Here synapsedata is my primary file system for the synapse workspace.

enter image description here

To store the files in the mount point other than the primary file system, we can use the path like synfs:/<jobid>/<mountpoint>/<path>.

Use the below code which is working for me to achieve your requirement.

jobid=mssparkutils.env.getJobId()
path='synfs:/'+jobid+'/devbronze/account'
print(path)

df1.write.format("delta").mode("overwrite").save(path)

First get the job id from mssparkutils and build the path with your mount location. Here devbronze is my mount point and container name which was created by using linked service.

enter image description here

Code Execution:

enter image description here

Result files in mount path:

enter image description here