Search code examples
amazon-sagemaker

Sagemaker custom model metrics from eval pipeline step

I've created a sagemaker pipeline with a register model step show below. The model_metrics is produced by an eval step. It's based of the sagemaker tutorials but all the examples I can find use an SKLearn estimator, my model is pytorch based (it's a sentence-transformers model evaluated using their information retrieval evaluator).

The issue I have is when I view the registered model in sagemaker studio (latest version, not sagemaker studio classic). I'm expecting metrics to appear on the "Performance" tab under "Model Metrics" but all I get is a spinner then "NetworkError when attempting to fetch resource."

This didn't happen before I added model_metrics to model.register so I'm assuming that metrics are supposed to appear here? I've checked that the file exists at the uri it's being registered at.

Has anyone got this working? Does the file have to be a specific format? I tried to upload an example evaluation.json file from the tutorial and passed the uri to it but I got the same error.

Is there a way of getting a more detailed error message?

enter image description here

model_metrics = ModelMetrics(
    model_statistics=MetricsSource(
        s3_uri=Join(
            on="/",
            values=[
                evaluation_step.properties.ProcessingOutputConfig.Outputs["evaluation"].S3Output.S3Uri,
                "evaluation.json",
            ],
        ),
        content_type="application/json",
    )
)

register_args = model.register(
    content_types=["application/json"],
    response_types=["application/json"],
    transform_instances=["ml.g4dn.xlarge"],  # g5's not available for batch transform?
    inference_instances=["ml.g4dn.xlarge", "ml.g5.xlarge"],
    model_package_group_name=model_package_group_name,
    approval_status="PendingManualApproval",
    model_metrics=model_metrics,
)

register_model_step = ModelStep(name="RegisterModel", step_args=register_args)
Solution

  • It seems that your S3 bucket lacks the necessary CORS policy required for the new SageMaker Studio.

    In the new Studio experience, S3 buckets must have a CORS configuration applied because Studio accesses S3 via the browser, which utilizes CORS to handle cross-origin requests. Without the proper CORS policy, requests to S3 will not succeed.

    If you migrated from Studio Classic to the new version, it's likely that the default S3 bucket for SageMaker does not have the appropriate CORS configuration.

    Detailed instructions for adding the CORS configuration can be found here.

    In a nutshell, you need to add the following CORS policy to the desired S3 bucket(s):

    [
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "POST",
            "PUT",
            "GET",
            "HEAD",
            "DELETE"
        ],
        "AllowedOrigins": [
            "https://*.sagemaker.aws"
        ],
        "ExposeHeaders": [
            "ETag",
            "x-amz-delete-marker",
            "x-amz-id-2",
            "x-amz-request-id",
            "x-amz-server-side-encryption",
            "x-amz-version-id"
        ]
    }
]