Search code examples
azure-machine-learning-service

Azure Machine Learning Designer Error: JobConfigurationMaxSizeExceeded

I have an Azure Machine Learning Designer pipeline that I've run successfully many dozens of times. Suddenly, today, The pipeline is getting down to the 'Train Model' node and failing with the following error:

JobConfigurationMaxSizeExceeded: The specified job configuration exceeds the max allowed size of 32768 characters. Please reduce the size of the job's command line arguments and environment settings

How do I address this error in designer-built pipelines?

I have even gone back to previously successful runs of this pipeline and resubmitted one of these runs which also failed with the exact same error. A resubmitted run should have the exact same pipeline architecture and input data (afaik), so it seems like a problem outside my control.

Pipeline with error: enter image description here

Pipeline run overview: enter image description here Any ideas?

EDIT: I'm able to repro this with a really simple pipeline. Simply trying to exclude columns in a Select Columns node from a dataset gives me this error:

enter image description here

Solution

  • This appears to be a bug introduced by Microsoft's rollout of their new Compute Common Runtime.

    If I go into any nodes failing with the JobConfigurationMaxSizeExceeded exception and manually set AZUREML_COMPUTE_USE_COMMON_RUNTIME:false in their Environment JSON field, then they work correctly.