My Question(s)
As there is a bit of code to back up this question - I'll ask it upfront. Are there any known performance issues with a Servicestack self host service (or indeed any http listener) running on linux/mono?
My actual use case is for a web service calling multiple other (non-public) web services. When running under windows I notice that the performance is blindingly fast, but when running under linux/mono - it seems to slow down and the length of a request can take up to 15 seconds (compared to 0.2 seconds running under windows).
My follow up question is - what (if anything) am I doing incorrectly here?
.
My Environment(s)
I am running a Windows 10 PC - i7-6700 @ 3.4ghz 4 cores - (hyperthreaded - so 8 logical cores), 32GB ram, and have a Linux VM (Ubuntu 16.04) using hyper V. It has 2 cores ([email protected] - 4GB Ram assigned to it). Basically - nothing in the code below should be over-taxing the hardware underlying the service definition below. I have also tried this hosted on other VM's to make sure it wasn't my local hardware - but I seem to get consistent results wherever I try.
I use the mono:latest image and xbuild my C# solution to create a docker image which I host on a the linux machine.
Also - I am pretty new to the world of Linux - not really sure how to troubleshoot on this platform (yet!)
An example of one of the services
Program class: works with both windows / linux:
class Program
{
static void Main(string[] args)
{
var listeningOn = args.Length == 0 ? "http://*:32700/api/user/" : args[0];
var appHost = new AppHost()
.Init()
.Start(listeningOn);
Console.WriteLine("AppHost Created at {0}, listening on {1}",
DateTime.Now, listeningOn);
// check if we're running on mono
if (Type.GetType("Mono.Runtime") != null)
{
// on mono, processes will usually run as daemons - this allows you to listen
// for termination signals (ctrl+c, shutdown, etc) and finalize correctly
UnixSignal.WaitAny(new[]
{
new UnixSignal(Signum.SIGINT),
new UnixSignal(Signum.SIGTERM),
new UnixSignal(Signum.SIGQUIT),
new UnixSignal(Signum.SIGHUP)
});
}
else
{
Console.ReadLine();
}
}
}
App host:
public class AppHost : AppSelfHostBase
{
public AppHost() : base("Test User Service", typeof(AppHost).Assembly)
{
Plugins.Add(new PostmanFeature());
Plugins.Add(new CorsFeature());
}
public override void Configure(Container container)
{
}
}
Contracts:
[Api("Get User"), Route("/getUserByUserIdentifier/{Tenant}/{Identifier}", "GET")]
public class GetUserByUserIdentifierRequest : IReturn<GetUserByUserIdentifierResponse>
{
public string Tenant { get; set; }
public string Identifier { get; set; }
}
public class GetUserByUserIdentifierResponse
{
public UserDto User { get; set; }
}
public class UserDto
{
public string UserName { get; set; }
public string UserIdentifier { get; set; }
}
Separate console that I use to test the application:
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Starting...");
ConcurrentBag<long> timings = new ConcurrentBag<long>();
Parallel.For(0, 100, new ParallelOptions { MaxDegreeOfParallelism = 8 }, x =>
{
Console.WriteLine("Attempt #{0}", x);
using (JsonServiceClient client = new JsonServiceClient("http://127.0.0.1:32700/api/user/")) //Specify Linux box IP here!
{
Stopwatch sw = new Stopwatch();
Console.WriteLine("Stopwatch started...");
sw.Start();
GetUserByUserIdentifierResponse response = client.Get(new GetUserByUserIdentifierRequest {Tenant = "F119A0DF-5002-4FF1-A0CE-8B60CFEE16A2", Identifier = "3216C49E-80C9-4249-9407-3E636E8C58AC"});
sw.Stop();
Console.WriteLine("Stopwatch stopped... got value [{0}] back in {1}ms", response.ToJson(), sw.ElapsedMilliseconds);
timings.Add(sw.ElapsedMilliseconds);
}
});
var allTimes = timings.ToList();
Console.WriteLine("Longest time taken = {0}ms", allTimes.Max());
Console.WriteLine("Shortest time taken = {0}ms", allTimes.Min());
Console.WriteLine("Avg time taken = {0}ms", allTimes.Average());
Console.WriteLine("Done!");
Console.ReadLine();
}
}
Results:
So on my local windows box, this can take wetween 0.1 of a second to 0.02 of a second per request. After multple attempts it averages out at around 0.1 of a second for 100 requests.
If I point the test application at the linux box - to the same code which is compiled with and running under mono in a docker container - I see that most of the requests are answered between 0.8 of a second and 0.05 of a second, but I do see (nearly every time I try) that some requests take 15 seconds to be serviced. This is a lot slower on Mono/linux than it is on .NET/Windows!
As an aside, if, i increase the parallel loop from 100 to 500, I find that the windows service handles all the requests without breaking a sweat, but the client application (my test program) will fail with an IO error:
System.IO.IOException was unhandled by user code
HResult=-2146232800
Message=Unable to read data from the transport connection: The connection was closed.
Source=System
StackTrace:
at System.Net.ConnectStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadToEnd()
at ServiceStack.Text.JsonSerializer.DeserializeFromStream[T](Stream stream)
at ServiceStack.Serialization.JsonDataContractSerializer.DeserializeFromStream[T](Stream stream)
at ServiceStack.JsonServiceClient.DeserializeFromStream[T](Stream stream)
at ServiceStack.ServiceClientBase.GetResponse[TResponse](WebResponse webResponse)
at ServiceStack.ServiceClientBase.Send[TResponse](String httpMethod, String relativeOrAbsoluteUrl, Object request)
at ServiceStack.ServiceClientBase.Get[TResponse](IReturn`1 requestDto)
at httppoke.Program.<>c__DisplayClass0_0.<Main>b__0(Int32 x) in <Redacted>\Program.cs:line 30
at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
InnerException:
I have a feeling that this error may help indicate what is going on, but I don't really know how to interperate it in the context of the problem I am facing.
It is also worth noting, that watching either 'top' or 'docker stats' on the linux machine while the test program is running, the CPU usage never goes above 4%. The service isn't exactly doing anything taxing.
Please note - I am trying to get multiple services talking to each other - the service shown here is a very cut down version of a 'test user' service. I find that when the services call other services (each service runs in it's own docker container) - the amount of time it takes to communicate between the services in unacceptably long.
The reason I am only showing you all one service here is that it is demonstrable that multiple calls to a service appear to slow down noticeably.
I'm not sure it is a problem being caused by service stack, as I also have a self hosted Nancyfx service running, and the same behaviour is apparent with that service too. Help!
ServiceStack added support for .NET Core in its v4.5.2 Release which is now the recommended and supported option for running ServiceStack on Linux.
Are there any known performance issues with a Servicestack self host service (or indeed any http listener) running on linux/mono?
Mono's HTTP Stack is slow and unstable for heavy workloads, it's fine for small workloads but we don't recommend it for production workloads. We've documented the most reliable setup we've found to run on Mono using HyperFastCI + nginx, e.g:
The future of hosting ServiceStack and .NET on Linux is .NET Core which is fast, stable and well supported. ServiceStack's support for .NET Core will be announced in the next v4.5.2 Release Notes to be announced later this week.