403 Forbidden when accessing Kroger API from Azure-Hosted .Net Core 3.1 app

Question

Problem: I need to determine the nature of this failure so I can know how to further troubleshoot. I have come up with a few hypothesis: - could be a firewall/proxy configuration within Azure - could be a misconfiguration with Kroger's API - could be a public certificate rejection from Azure app - could be something totally unrelated to any of above

Details: I’m attempting to connect to Kroger's developer API. The following code has been simplified for this post. (I have been using IHttpClientFactory previously to generate my HTTPClient). This works locally, but once it’s deployed to an Azure web service, I am presented with a 403 message (which appears to be coming from Azure and not the external API):

You don't have permission to access http://api.kroger.com on this server.

This same code works within Azure for other 3rd party APIs over HTTPS, so the suspicion I have had is that this error comes from not using correct client certificates, so Azure attempts to call over http?

I have tried many things to resolve this, including uploading a public certificate that I downloaded from https://api.kroger.com to my azure app service. It appears to pull the certificate correctly, but the request still fails with the same message.

Relevant code is below (Without using client certificate):

using(var client = _requestFactory.CreateClient()))
{
        client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("basic", _clientId);
        client.DefaultRequestHeaders.Add("Accept", "application/json");

        var newRequest = new Dictionary<string, string>
        {
            { "grant_type", "client_credentials" },
            { "scope", "product.compact" }
        };

        var response = await client.PostAsync($"https://api.kroger.com/v1/connect/oauth2/token", new FormUrlEncodedContent(newRequest));
        return Ok(await response.Content.ReadAsStringAsync());
}

Below is the full response from the server.

{
    "version": "1.1",
    "content": {
        "headers": [{
            "key": "Content-Type",
            "value": ["text/html"]
        }, {
            "key": "Content-Length",
            "value": ["299"]
        }, {
            "key": "Expires",
            "value": ["Sat, 08 Feb 2020 19:18:55 GMT"]
        }]
    },
    "statusCode": 403,
    "reasonPhrase": "Forbidden",
    "headers": [{
        "key": "Server",
        "value": ["AkamaiGHost"]
    }, {
        "key": "Mime-Version",
        "value": ["1.0"]
    }, {
        "key": "Date",
        "value": ["Sat, 08 Feb 2020 19:18:55 GMT"]
    }, {
        "key": "Connection",
        "value": ["close"]
    }, {
        "key": "Set-Cookie",
        "value": ["akaalb_Digital_ALB_API=~op=KT_Digital_API_KCVG_F5:api-kcvg|~rv=47~m=api-kcvg:0|~os=75b4a9ec926d2a9e67035451773cec6c~id=63ba4b3e2a027e4d53b693e2fded5ac3; path=/; HttpOnly; Secure; SameSite=None"]
    }],
    "trailingHeaders": [],
    "requestMessage": {
        "version": "1.1",
        "content": {
            "headers": [{
                "key": "Content-Type",
                "value": ["application/x-www-form-urlencoded"]
            }, {
                "key": "Content-Length",
                "value": ["51"]
            }]
        },
        "method": {
            "method": "POST"
        },
        "requestUri": "https://api.kroger.com/v1/connect/oauth2/token",
        "headers": [{
            "key": "Authorization",
            "value": ["basic {removed}"]
        }, {
            "key": "Accept",
            "value": ["application/json"]
        }, {
            "key": "Request-Context",
            "value": ["appId={removed}"]
        }, {
            "key": "Request-Id",
            "value": ["|{removed}"]
        }, {
            "key": "traceparent",
            "value": ["{removed}"]
        }],
        "properties": {}
    },
    "isSuccessStatusCode": false
}

Answer 1

"headers": [{ "key": "Server", "value": ["AkamaiGHost"] }

Almost certainly Akamai issue. I have seen 403 errors due to header order or even user-Agent header that Akamai does not like.