Search code examples
pythonazuregzipazure-machine-learning-service

AzureML Experiment doesnt finish (Complete) - outputs exists


I have a pythion script which I run as:

config = ScriptRunConfig(
    source_directory='logic', 
    script='sense/myscript.py',
    arguments = ['--val', 'decode'],
    compute_target=target,
    environment=myenv_local,
)
# submit script to AML
run = exp.submit(config)

The script which I run has these steps at the end of script:

print('---- Fitting pipeline')
pipeline.fit(X_train,y_train)

print('Saving joblibs')
    
dill.dump(pipeline, gzip.open('./outputs/baseModel_LR_MONTH0_5D.gz',"wb"))
print('---- Finished.')

The output in azureml looks like this:

---- Fitting pipeline
Saving joblibs

But it never prints the "---- Finished." and shows that the job is running: enter image description here

The model I save to output is there in the run and I can download it: enter image description here


Solution

  • The problem was it was still processing. Thee job lib being produced was about 27GB that's why it looked like it was stuck but actually, it was writing files.