How to write Azure machine learning batch scoring results to data lake?

Question

I'm trying to write the output of batch scoring into datalake:

    parallel_step_name = "batchscoring-" + datetime.now().strftime("%Y%m%d%H%M")
    
    output_dir = PipelineData(name="scores", 
                              datastore=def_ADL_store,
                              output_mode="upload",
                              output_path_on_compute="path in data lake")

parallel_run_config = ParallelRunConfig(
    environment=curated_environment,
    entry_script="use_model.py",
    source_directory="./",
    output_action="append_row",
    mini_batch_size="20",
    error_threshold=1,
    compute_target=compute_target,
    process_count_per_node=2,
    node_count=2
)
    
    batch_score_step = ParallelRunStep(
        name=parallel_step_name,
        inputs=[test_data.as_named_input("test_data")],
        output=output_dir,
        parallel_run_config=parallel_run_config,
        allow_reuse=False
    )

However I meet the error: "code": "UserError", "message": "User program failed with Exception: Missing argument --output or its value is empty."

How can I write results of batch score to data lake?

Answer 1

I don’t think ADLS is supported for PipelineData. My suggestion is to use the workspace’s default blob store for the PipelineData, then use a DataTransferStep for after the ParallelRunStep is completed.