I’m using the following script to execute an AutoML run, also passing the test dataset
automl_settings = {
"n_cross_validations": 10,
"primary_metric": 'spearman_correlation',
"enable_early_stopping": True,
"max_concurrent_iterations": 10,
"max_cores_per_iteration": -1,
"experiment_timeout_hours": 1,
"featurization": 'auto',
"verbosity": logging.INFO}
automl_config = AutoMLConfig(task = 'regression',
debug_log = 'automl_errors.log',
compute_target = compute_target,
training_data = training_data,
test_data = test_data,
label_column_name = label_column_name,
model_explainability = True,
**automl_settings )
Note that the TEST DATASET SUPPORT is a feature still in PRIVATE PREVIEW. It'll probably be released as PUBLIC PREVIEW later in NOVEMBER, but until then, you need to be enrolled in the PRIVATE PREVIEW in order to see the "Test runs and metrics" in the UI. You can send me an email to cesardl at microsoft dot com and send me your AZURE SUBSCRIPTION ID to be enabled so you see it in the UI.
You can see further info on how to get started here: https://github.com/Azure/automl-testdataset-preview
About how to use it, you need to either provide the test_Data (specific Test AML Tabular Dataset that for instance you loaded from a file os split manually previously) or you can provide a test_size which is the % (i.e. 0.2 is 20%) to be split from the single/original dataset.
About the TEST metrics, since you can make multiple TEST runs against a single model, you need to go to the specific TEST run available under the link "Test results"
