According to the documentation, we should be able to add custom libraries as follows:
az synapse spark pool update --name testpool \
--workspace-name testsynapseworkspace --resource-group rg \
--package-action Add --package package1.jar package2.jar
However, when I try this with my python package whl files, I get an error message that the package does not exist.
> $new_package_names = "PACKAGE1-1.0.1-py3-none-any.whl PACKAGE2-1.0.6.3-py3-none-any.whl PACKAGE3-1.0.0-py3-none-any.whl PACKAGE4-1.0.1-py3-none-any.whl"
> az synapse spark pool update --name $pool_name --workspace-name $workspace_name --resource-group $resource_group --package-action Add --package $new_package_names
I receive the following error:
(LibraryDoesNotExistInWorkspace) The LibraryArtifact PACKAGE1-1.0.1-py3-none-any.whl PACKAGE2-1.0.6.3-py3-none-any.whl PACKAGE3-1.0.0-py3-none-any.whl PACKAGE4-1.0.1-py3-none-any.whl does not exist.
Code: LibraryDoesNotExistInWorkspace
Message: The LibraryArtifact PACKAGE1-1.0.1-py3-none-any.whl PACKAGE2-1.0.6.3-py3-none-any.whl PACKAGE3-1.0.0-py3-none-any.whl PACKAGE4-1.0.1-py3-none-any.whl does not exist.
The same works if I have only one package in the variable $new_package_names.
It looks to me like Azure thinks it's all one package instead of four different ones. All four are uploaded to the synapse workspace and available for selection when I do the same process manually. Does anyone know of a fix for this issue? Does it only work for .jar files for some reason?
Turns out that it really comes down to the format in which I pass the package names to the function. Something apparently changed internally as the previous way did not work anymore.
As MartinJaffer from Microsoft answered in the MS Q&A forum:
"""
If you are using az in powershell, there is a better way to go about this.
$new_package_names = "PACKAGE1-1.0.1-py3-none-any.whl" , "PACKAGE2-1.0.6.3-py3-none-any.whl" , "PACKAGE3-1.0.0-py3-none-any.whl" , "PACKAGE4-1.0.1-py3-none-any.whl"
az synapse spark pool update --name $pool_name --workspace-name $workspace_name --resource-group $resource_group --package-action Add --package @new_package_names
Here we changed new_package_names into an array type, and use the @ splatter operator to seperate them.
As simpler example, it makes the following two excerpts be equivalent:
Copy-Item "test.txt" "test2.txt" -WhatIf
$ArrayArguments = "test.txt", "test2.txt"
Copy-Item @ArrayArguments -WhatIf
"""
Utilizing the splatter operator when passing the parameters worked perfectly.