Search code examples
c#azureazure-blob-storageparquet

Unable to send blob to Azure - missing HTTP header exception

I am trying to upload a parquet file to Azure, but it keeps giving me an error message saying that its missing a required HTTP header.

In the example I am providing I just tried some basic data in a memory stream and it is still throwing this exception.

RequestFailedException: An HTTP header that's mandatory for this request is not specified. RequestId:0bd87df1-701f-0003-5ff4-944ccd000000 Time:2024-04-22T20:36:16.2840714Z Status: 400 (An HTTP header that's mandatory for this request is not specified.) ErrorCode: MissingRequiredHeader

Content: {"error":{"code":"MissingRequiredHeader","message":"An HTTP header that's mandatory for this request is not specified.\nRequestId:...\nTime:2024-04-22T20:36:16.2840714Z"}}

Headers: Connection: keep-alive x-ms-error-code: MissingRequiredHeader x-ms-request-id: ... x-ms-version: 2024-05-04 x-ms-client-request-id: ... Content-Length: 204 Content-Type: application/json;charset=utf-8 Date: Mon, 22 Apr 2024 20:36:16 GMT Server: Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0 Via: REDACTED

Code:

            string storageAccountName = "name";
            string storageAccountKey = "key";
            StorageSharedKeyCredential sharedKeyCredential = new StorageSharedKeyCredential(storageAccountName, storageAccountKey);

            BlobServiceClient blobServiceClient = new BlobServiceClient(
                new Uri("https://" + storageAccountName + ".dfs.core.windows.net"),
                sharedKeyCredential
            );

            BlobContainerClient desContainer = blobServiceClient.GetBlobContainerClient("testcontainer");

            BlobClient desBlob = desContainer.GetBlobClient("/testAPI/my.parquet");

            var myStr = "Hello!";
            var content = Encoding.UTF8.GetBytes(myStr);
            using (var ms = new MemoryStream(content))
                desBlob.Upload(ms);

I am not able to find what headers are missing and I am not sure how to add them as there does not seem a standard way to add headers besides; Content-Type

Packages:

  • Azure.Core - 1.39.0
  • Azure.Storage.Blobs - 12.20.0-beta.2
  • Azure.Storage.Common - 12.19.0-beta.2

I've tried reverting to older packages thinking maybe functionality was there before but changed in recent revisions.

Any help in the right direction would be helpful.

Solution

  • As mentioned in the comments, please change the endpoint suffix from dfs.core.windows.net to blob.core.windows.net. My guess is that when the REST API sees dfs.core.windows.net endpoint, it tries to use Data Lake REST API instead of Blob Storage REST API and because of that, you are getting the missing header error.