Sagemaker Pipeline Error: Failed to upload to jumpstart-cache-prod-ap-southeast-1/source-directory-tarballs/lightgbm/inference/classification

I encountered this error when running pipeline.upsert()


S3UploadFailedError: Failed to upload /tmp/tmpexrxqr32/new.tar.gz to jumpstart-cache-prod-ap-southeast-1/source-directory-tarballs/lightgbm/inference/classification/v1.1.2/sourcedir.tar.gz: An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: Access Denied

My pipeline consists of preprocessing, training, evaluating, creating model and transforming step. When i ran these steps seprarately they were working just fine, but when I put them together in a pipeline, the mentioned error occured. Can anyone tell me what is the cause of this error, I did not write any line of code to upload anything to Jumpstart S3.

model = Model(
    image_uri=infer_image_uri,
    model_data=step_train.properties.ModelArtifacts.S3ModelArtifacts,
    sagemaker_session=pipeline_session,
    role=role,
    source_dir=infer_source_uri,
    entry_point="inference.py"
)

When I comment out the entry_point line, pipeline.upsert() returned no error, but the transform job failed. The model I'm using is JumpStart LightGBM.

Solution

Nature of the problem

This happens by default because your experiment tries to upload the source code into its default bucket (which is the jumpstart bucket).

You can check the default bucket assigned to pipeline_session by printing pipeline_session.default_bucket().

Evidently you do not have the correct write permissions on that bucket and I encourage you to check them.

When you comment the entry_point, it doesn't give you that error precisely because it doesn't have anything to load. However, the moment it tries to do inference, it does not find the script clearly.

One possible quick and controllable solution

If you want to apply a cheat to verify what I told you, try putting the code_location parameter in the Model.

This way you can control exactly where your pipeline step goes to write. You will clearly need to specify the s3 uri of the desired destination folder.

code_location (str) – Name of the S3 bucket where custom code is uploaded (default: None). If not specified, the default bucket created by sagemaker.session.Session is used.