How can I get the provisioning state of my Workspace Package to be "Succeeded" and not "Incomplete" when I create it through the SDK?

Question

I am using the ArtifactsClient from azure-synapse-artifacts library and trying to upload a new Workspace Package. When doing this manually through the UI it works and my Provisioning State is set to "Succeeded". When I try to do the same though the SDK I won't get the "Succeeded", and hence cannot use the library in my Spark Pool.

Here is the code I'm using:

from time import sleep

from azure.identity import DefaultAzureCredential
from azure.synapse.artifacts import ArtifactsClient


client = ArtifactsClient(credential=DefaultAzureCredential(), endpoint="https://mysynapseworkspace.dev.azuresynapse.net")

ws = client.workspace.get()

library_client = client.library
wheel_name = 'dalib.whl'

poller = library_client.begin_create(wheel_name)
while not poller.done():
    print(poller.status())
    sleep(1)

whl = open('C:\\path\\to\\wheelfile\\wheel_tester-0.0.1-py3-none-any.whl', 'rb')
library_client.append(library_name=wheel_name, content=whl)
whl.close()

Answer 1

Ok so I solved this.

It seems that after you've created the library and called the append() method you also have to use begin_flush() in order to set the provisioning state to "succeeded". However, this is not mentioned in any documentation I have found. If anyone has the same problem that I had I post the working code below:

from time import sleep

from azure.core.exceptions import ResourceExistsError
from azure.identity import DefaultAzureCredential
from azure.synapse.artifacts import ArtifactsClient

credential = DefaultAzureCredential()

client = ArtifactsClient(credential=credential, endpoint="https://synws.dev.azuresynapse.net")

ws = client.workspace.get()


library_client = client.library
wheel_name = 'wheel_tester-0.0.1-py3-none-any.whl'
whl = open(f'C:\\path\\to\\wheelfile\\{wheel_name}', 'rb')

try:
    print('creating library')
    poller = library_client.begin_create(wheel_name)
    while not poller.done():
        print(poller.status())
        sleep(1)
    print(poller.status())
    print(poller.result())
except ResourceExistsError as ree:
    print('resource already exists.')

print('appending content')
library_client.append(library_name=wheel_name, content=whl)

print('flushing library')
lro_poller = library_client.begin_flush(library_name=wheel_name)
while not lro_poller.done():
    print(lro_poller.status())
    sleep(1)

print(lro_poller.status())
print(lro_poller.result())

whl.close()