Search code examples
azureazure-storageazure-machine-learning-serviceazure-blob-storage

Azure ML studio export data Azure Storage V2

I already post my problem here and they suggested me to post it here. I am trying to export data from Azure ML to Azure Storage but I have this error:

Error writing to cloud storage: The remote server returned an error: (400) Bad Request.. Please check the url. . ( Error 0151 )

My blob storage configuration is Storage v2 / Standard and Require secure transfer set as enabled.

If I set the Require secure transfer set as disabled, the export works fine.

How can I export data to my blob storage with the require secure transfer set as enabled ?

Solution

  • According to the offical tutorial Export to Azure Blob Storage, there are two authentication types for exporting data to Azure Blob Storage: SAS and Account. The description for them as below.

    1. For Authentication type, choose Public (SAS URL) if you know that the storage supports access via a SAS URL.

      A SAS URL is a special type of URL that can be generated by using an Azure storage utility, and is available for only a limited time. It contains all the information that is needed for authentication and download.

      For URI, type or paste the full URI that defines the account and the public blob.

    2. For private accounts, choose Account, and provide the account name and the account key, so that the experiment can write to the storage account.

      • Account name: Type or paste the name of the account where you want to save the data. For example, if the full URL of the storage account is http://myshared.blob.core.windows.net, you would type myshared.

      • Account key: Paste the storage access key that is associated with the account.

    I try to use a simple module combination as the figure and Python code below to test the issue you got.

    enter image description here

    import pandas as pd

def azureml_main(dataframe1 = None, dataframe2 = None):
    dataframe1 = pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]})
    return dataframe1,

    When I tried to use the authentication type Account of my Blob Storage V2 account, I got the same issue as yours which the error code is Error 0151 as below via click the View error log Button under the link of View output log.

    enter image description here

    Error 0151

    There was an error writing to cloud storage. Please check the URL.

    This error in Azure Machine Learning occurs when the module tries to write data to cloud storage but the URL is unavailable or invalid.

    Resolution Check the URL and verify that it is writable.

    Exception Messages

    • Error writing to cloud storage (possibly a bad url).
    • Error writing to cloud storage: {0}. Please check the url.

    Based on the error description above, the error should be caused by the blob url with SAS incorrectly generated by the Export Data module code with account information. May I think the code is old and not compatible with the new V2 storage API or API version information. You can report it to feedback.azure.com.

    However, I switched to use SAS authentication type to type a blob url with a SAS query string of my container which I generated via Azure Storage Explorer tool as below, it works fine.

    Fig 1: Right click on the container of your Blob Storage account, and click the Get Shared Access Signature

    enter image description here

    Fig 2: Enable the permission Write (recommended to use UTC timezone) and click Create button

    enter image description here

    Fig 3: Copy the Query string value, and build a blob url with a container SAS query string like https://<account name>.blob.core.windows.net/<container name>/<blob name><query string>

    enter image description here

    Note: The blob must be not exist in the container, otherwise an Error 0057 will be caused.