Getting IIS unauthorized html page instead of web api unauthorized response for unauthorized status

Question

I have a Web API application hosted on IIS 10 (Windows Server 2016). The application has an API for login. The login API is decorated with an custom Authentication Filter Attribute that implements IAuthenticationFilter.

[RoutePrefix("api/login")]
[AllowUnAuthorized]
[AuthenticationFilter]

public class LoginController : ApiController
{

    [HttpPost]
    public IHttpActionResult Login()
    {
        //Code to return token if passed authentication in the Authentication Filter
    }
 }   

If the login credentials (UserName and Password) are invalid, the Authentication Filter attribute sets an ErrorResult on the context that returns an status code of 401 ("Unauthorized") with a response message.

public class AuthenticationFilter : Attribute, IAuthenticationFilter
{
    public bool AllowMultiple => false;

    public async Task AuthenticateAsync(HttpAuthenticationContext context, 
    CancellationToken cancellationToken)
    {
        HttpRequestMessage request = context.Request;
        AuthenticationHeaderValue authorization = 
         request.Headers.Authorization;

        if (authorization == null)
        {
            return;
        }
        if (authorization.Scheme != "Basic")
        {
            return;
        }


        Tuple<string, string> credentials = 
        ExtractCredentials(request.Headers);
        if (credentials == null)
        {
            context.ErrorResult = new AuthenticationFailureResult("Invalid 
            credentials", request);                
            return;
        }
        string userName = credentials.Item1;
        string password = credentials.Item2;


        IAuthenticationService 
        service=container.Resolve<IAuthenticationService>();
        var user = service.GetUser(userName, password);
        if (user ==  null)
        {
            context.ErrorResult = new AuthenticationFailureResult("Invalid username or password", request);                
            return;
        }

        //Code to set principal if authentication passed
}

This is the code of the AuthenticationFailureResult class.

internal class AuthenticationFailureResult : IHttpActionResult
{
    public string ReasonPhrase { get; private set; }

    public HttpRequestMessage Request { get; private set; }

    public AuthenticationFailureResult(string reasonPhrase, HttpRequestMessage request)
    {
        ReasonPhrase = reasonPhrase;
        Request = request;
    }

    public Task<HttpResponseMessage> ExecuteAsync(CancellationToken cancellationToken)
    {
        return Task.FromResult(Execute());
    }

    private HttpResponseMessage Execute()
    {
        HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.Unauthorized);
        response.RequestMessage = Request;
        response.ReasonPhrase = ReasonPhrase;
        return response;
    }
}

This code was working fine and returning a 401 status code along with message as specified in the reason phrase. However I don't know due to what change, suddenly the API is returning an html page which is normally returned by IIS, with the message "401 - Unauthorized: Access is denied due to invalid credentials." instead of the error response set in the Authentication filter. Note that the same API works as expected when running in IISExpress

What I have done so far :

1. Checked Web.config and verified that CustomErrors Mode is set to "On"
2. Made sure that "Anonymous Authentication" is enabled in the "Authentication" section for the site in IIS

3. Added following to Web.config

   <system.webServer>
     <modules runAllManagedModulesForAllRequests="true"/>
   </system.webServer>
   

Strangely, the web api returns proper JSON response when authentication passes

Edit 1:

Here is the ChallengeAsync method

public Task ChallengeAsync(HttpAuthenticationChallengeContext context, CancellationToken cancellationToken)
        {
            var challenge = new AuthenticationHeaderValue("Basic");
            context.Result = new AddChallengeOnUnauthorizedResult(challenge, context.Result);
            return Task.FromResult(0);
        }

Implementation of AddChallengeOnUnauthorizedResult

public class AddChallengeOnUnauthorizedResult : IHttpActionResult
    {
        public AddChallengeOnUnauthorizedResult(AuthenticationHeaderValue challenge, IHttpActionResult innerResult)
        {
            Challenge = challenge;
            InnerResult = innerResult;
        }

        public AuthenticationHeaderValue Challenge { get; private set; }

        public IHttpActionResult InnerResult { get; private set; }

        public async Task<HttpResponseMessage> ExecuteAsync(CancellationToken cancellationToken)
        {
            HttpResponseMessage response = await InnerResult.ExecuteAsync(cancellationToken);

            if (response.StatusCode == HttpStatusCode.Unauthorized)
            {
                // Only add one challenge per authentication scheme.
                if (!response.Headers.WwwAuthenticate.Any((h) => h.Scheme == Challenge.Scheme))
                {
                    response.Headers.WwwAuthenticate.Add(Challenge);
                }
            }

            return response;
        }
    }

Answer 1

The solution was the add below two settings in Web.config.

 <system.webServer>
            <httpErrors existingResponse="PassThrough" />
            <modules runAllManagedModulesForAllRequests="true" />
  </system.webServer>