UPDATE 4 Even More confused I thought I was onto something, but the same error keeps popping up. This solution seemed promising, but isn't 100%, but thought it might help someone narrow down what's wrong.
Steps:
I was able to repeat that 4 times before I would still get the Error when hitting QA. After restarting this process again, I can now get my QA environment to work without having to use the API locally. Any ideas what's going on? Seems to be a setup issue/proxy/certificate issue, but no idea how to debug that.
This is the error that I'm seeing now: Server Error in '/' Application. An existing connection was forcibly closed by the remote host Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. Stack Trace:
[SocketException (0x2746): An existing connection was forcibly closed by the remote host]
System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) +8156963
System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) +48
[IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.]
System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult) +8111720
System.Net.TlsStream.EndRead(IAsyncResult asyncResult) +275
System.Net.Connection.ReadCallback(IAsyncResult asyncResult) +45
[WebException: The underlying connection was closed: An unexpected error occurred on a receive.]
System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) +764
System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar) +78
[HttpRequestException: An error occurred while sending the request.]
[AggregateException: One or more errors occurred.]
System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) +4451240
Microsoft.IdentityModel.Protocols.<GetDocumentAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\HttpDocumentRetriever.cs:53
[IOException: Unable to get document from: https://securityeliqa.twcable.com/core/.well-known/openid-configuration]
Microsoft.IdentityModel.Protocols.<GetDocumentAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\HttpDocumentRetriever.cs:59
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.IdentityModel.Protocols.<GetAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\OpenIdConnectConfigurationRetriever.cs:81
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.IdentityModel.Protocols.<GetConfigurationAsync>d__3.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\ConfigurationManager.cs:0
[InvalidOperationException: IDX10803: Unable to create to obtain configuration from: 'https://securityeliqa.twcable.com/core/.well-known/openid-configuration'.]
Microsoft.IdentityModel.Protocols.<GetConfigurationAsync>d__3.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\ConfigurationManager.cs:212
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.IdentityModel.Protocols.<GetConfigurationAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\ConfigurationManager.cs:0
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() +31
IdentityServer.WebApi.AccessTokenValidation.<<RetrieveMetadata>b__0>d__4.MoveNext() in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\DiscoveryDocumentIssuerSecurityTokenProvider.cs:123
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() +31
IdentityServer.WebApi.AccessTokenValidation.AsyncHelper.RunSync(Func`1 func) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\AsyncHelper.cs:18
IdentityServer.WebApi.AccessTokenValidation.DiscoveryDocumentIssuerSecurityTokenProvider.RetrieveMetadata() in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\DiscoveryDocumentIssuerSecurityTokenProvider.cs:141
IdentityServer.WebApi.AccessTokenValidation.DiscoveryDocumentIssuerSecurityTokenProvider..ctor(String discoveryEndpoint, IdentityServerBearerTokenAuthenticationOptions options, ILoggerFactory loggerFactory) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\DiscoveryDocumentIssuerSecurityTokenProvider.cs:43
Owin.IdentityServerBearerTokenValidationAppBuilderExtensions.ConfigureLocalValidation(IdentityServerBearerTokenAuthenticationOptions options, ILoggerFactory loggerFactory) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\IdentityServerBearerTokenValidationAppBuilderExtensions.cs:129
Owin.IdentityServerBearerTokenValidationAppBuilderExtensions.UseIdentityServerBearerTokenAuthentication(IAppBuilder app, IdentityServerBearerTokenAuthenticationOptions options) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\IdentityServerBearerTokenValidationAppBuilderExtensions.cs:39
Company.WebApi.waAddressQualification.Startup.Configuration(IAppBuilder app) in e:\Source Code\GitHub\waDemo\Startup.cs:23
[TargetInvocationException: Exception has been thrown by the target of an invocation.]
System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0
System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +128
System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +146
Owin.Loader.<>c__DisplayClass12.<MakeDelegate>b__b(IAppBuilder builder) +93
Owin.Loader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder) +209
Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup) +843
Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(Action`1 startup) +51
Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint() +101
System.Threading.LazyInitializer.EnsureInitializedCore(T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory) +137
Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context) +172
System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers) +618
System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context) +172
System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context) +402
System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext) +343
[HttpException (0x80004005): Exception has been thrown by the target of an invocation.]
System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +579
System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +112
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +712
Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.6.1055.0
UPDATE 3 Even More confused
At this point I believe it's the server that is cancelling my requests, but even with that I can't be 100% sure. Here are some additional data points:
Original Questions
I'd like to know if I'm call my Demo API correctly, or if the API is causing me an issue. If I send three requests through at the same time I get the issue on at least one of the addresses I'm trying to process.
Update Number one
In the Inner Exception I'm getting the following message, I was able to capture a more complete error as follows: "The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host"
I'm using .NET 4.6.1 for both client and server side I'm not using a proxy. I'm able to successfully call the API about 1/3 of the time
I've experimented with the couple of different options in calling the service, the first was just wrapping the DemoApiClient in a lock, the next was to use Task Run with the async and await keywords and finally retrieving the token for authentication using Task Run or just calling it and having ConfigureAwait(false) listed on the await lines. I'm not sure if any of this matters, or if the server is just failing when called. The server isn't that complex, but I'd rather not post that code here. It does validate the token and then process an address, makes a couple of database calls and returns data found from the database.
UPDATE 2 I test with three addresses at a time, when I send them through SoapUI to the server all three are process fine, but with the .Net C# client using the code below only one of the three requests seem to make it to the server. I was expecting to see two stacktraces and one successful request, but the IIS log only shows one request coming in.
The code I'm using to call the API is here:
using Demo.IdentityServer.IdentityModel.Client;
using Newtonsoft.Json;
using PartnerPortal.Model;
using System;
using System.Configuration;
using System.Diagnostics;
using System.Net.Http;
using System.Linq;
using System.Collections.Generic;
using System.Threading.Tasks;
namespace DemoPortal.Client
{
public static class DemoApiClient
{
public readonly static string waDemoUrl = ConfigurationManager.AppSettings["waDemoURL"];
public static async Task<DemoData> GetDemoData(RequestAddress requestAddress)
{
try
{
using (var httpClient = new HttpClient() { MaxResponseContentBufferSize = 10000000, Timeout = TimeSpan.FromMilliseconds(5000) })
{
string reasonPhrase = "";
var demoUri = new Uri(string.Format(waDemoUrl + "api/DemoApp/GetSomeData?trackingId={0}&clientKey={1}&address={2}", requestAddress.TrackingId, requestAddress.ClientKey, requestAddress.Address));
var httpResult = new HttpResponseMessage();
var result = RequestAccessToken.RequestToken().Result;
var accessToken = await Task.Run(() => RequestAccessToken.RequestToken().Result);
//var accessToken = RequestAccessToken.RequestToken().Result;
//which of the above two options should I use? Does it matter?
httpClient.SetBearerToken(accessToken);
HttpResponseMessage response = await httpClient.GetAsync(demoUri);
HttpContent httpContent = response.Content;
if (httpResult.IsSuccessStatusCode)
{
var content = await httpContent.ReadAsStringAsync();
DemoData demoData = null;
demoData = JsonConvert.DeserializeObject<DemoData>(content);
return demoData;
}
else
{
reasonPhrase = httpResult.ReasonPhrase;
if (reasonPhrase.ToUpper() == "UNAUTHORIZED")
{
throw new KeyNotFoundException("Not authorized");
}
}
}
}
catch (Exception ex)
{
Debug.WriteLine("Message is:" + ex);
throw (ex);
}
return null;
}
}
}
And this is how the token is retrieved:
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using Thinktecture.IdentityModel.Tokens.Http;
namespace Demo.IdentityServer.IdentityModel.Client
{
public static class RequestAccessToken
{
public readonly static string securityUrl = ConfigurationManager.AppSettings["securityUrl"];
public readonly static string clientSecret = ConfigurationManager.AppSettings["clientSecret"];
public static async Task<string> RequestToken()
{
var url = new Uri(securityUrl);
var fields = new Dictionary<string, string>
{
{ OAuth2Constants.GrantType, OAuth2Constants.GrantTypes.ClientCredentials },
{ OAuth2Constants.Scope, "Read"}
};
using (var httpClient = new HttpClient())
{
var cancellationToken = new CancellationToken();
httpClient.DefaultRequestHeaders.Authorization = new BasicAuthenticationHeaderValue("Demo", clientSecret);
var ss = JsonConvert.SerializeObject(fields);
var data = new FormUrlEncodedContent(fields);
var s = data.ReadAsStringAsync();
//var response = await httpClient.PostAsync(url, data, cancellationToken).ConfigureAwait(false);
var response = await httpClient.PostAsync(url, data, cancellationToken);
// Should I use ConfigureAwait(false) or call method with a Task(Run() ... ?
if (response.StatusCode == HttpStatusCode.OK || response.StatusCode == HttpStatusCode.BadRequest)
{
//var content = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
var content = await response.Content.ReadAsStringAsync();
var token = new TokenResponse(content);
var accessToken = token.AccessToken;
return accessToken;
}
else
{
return new TokenResponse(response.StatusCode, response.ReasonPhrase).AccessToken;
}
}
}
}
}
SOLUTION Found!!! The entire issue stemmed from a misconfigured Load Balancer. Something with the SNAT pool. I'm not the network guy, and whatever he did fixed it. Something like the load balancer was referencing the box name directly instead of the SNAT Pool name, or the SNAT Pool had the wrong name. Either way once updated the intermittent successes became always successes.