I have a pythion script which I run as:
config = ScriptRunConfig(
source_directory='logic',
script='sense/myscript.py',
arguments = ['--val', 'decode'],
compute_target=target,
environment=myenv_local,
)
# submit script to AML
run = exp.submit(config)
The script which I run has these steps at the end of script:
print('---- Fitting pipeline')
pipeline.fit(X_train,y_train)
print('Saving joblibs')
dill.dump(pipeline, gzip.open('./outputs/baseModel_LR_MONTH0_5D.gz',"wb"))
print('---- Finished.')
The output in azureml looks like this:
---- Fitting pipeline
Saving joblibs
But it never prints the "---- Finished." and shows that the job is running:
The model I save to output is there in the run and I can download it:
The problem was it was still processing. Thee job lib being produced was about 27GB that's why it looked like it was stuck but actually, it was writing files.