I've recently started working gRPC, and always heard about it being much faster than REST. So I created I created a benchmark project, because I wanted to know how much faster it actually was. Turns out that after several different approaches, REST was always slightly better than gRPC. I'm not experienced neither with benchmarking nor gRPC, so something must be wrong. My guess is that maybe my benchmarking setup is not right.
So my question is: what is wrong with my benchmarking? Or do the results actually make sense?
I have created a repository with the complete code here: https://github.com/henridd/RestVsGrpcBenchmark. For this question, I'll be talking about the BenchmarkWithSamePayload run.
These are the results of the last test.
// * Summary *
BenchmarkDotNet v0.13.12, Windows 11 (10.0.22631.3155/23H2/2023Update/SunValley3)
AMD Ryzen 5 5600X, 1 CPU, 12 logical and 6 physical cores
.NET SDK 8.0.201
[Host] : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
DefaultJob : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
| Method | Mean | Error | StdDev |
|-------------- |---------:|--------:|--------:|
| BenchmarkGrpc | 147.2 us | 0.71 us | 0.59 us |
| BenchmarkRest | 111.6 us | 0.68 us | 0.90 us |
// * Hints *
Outliers
BenchmarkWithSamePayload.BenchmarkGrpc: Default -> 2 outliers were removed, 3 outliers were detected (145.56 us, 150.79 us, 152.36 us)
BenchmarkWithSamePayload.BenchmarkRest: Default -> 8 outliers were removed (118.18 us..126.86 us)
// * Legends *
Mean : Arithmetic mean of all measurements
Error : Half of 99.9% confidence interval
StdDev : Standard deviation of all measurements
1 us : 1 Microsecond (0.000001 sec)
You can find below the relevant code for this benchmark.
rpc SayHello (HelloRequest) returns (HelloReply);
message HelloRequest {
string name = 1;
}
message HelloReply {
string message = 1;
}
public class Sender
{
private GrpcChannel _grpcChannel;
private Greeter.GreeterClient _greeter;
private HelloRequest _defaultRequest;
public Sender()
{
_grpcChannel = GrpcChannel.ForAddress("http://localhost:5264");
_greeter = new Greeter.GreeterClient(_grpcChannel);
_defaultRequest = new HelloRequest() { Name = "default" };
}
public async Task<string> PostDefault()
{
var reply = await _greeter.SayHelloAsync(_defaultRequest);
return reply.Message;
}
}
using GrpcService.Services;
var builder = WebApplication.CreateBuilder(args);
// Add services to the container.
builder.Services.AddGrpc();
var app = builder.Build();
// Configure the HTTP request pipeline.
app.MapGrpcService<Service>();
app.Run();
public class Service : Greeter.GreeterBase
{
public override Task<HelloReply> SayHello(HelloRequest request, ServerCallContext context)
{
return Task.FromResult(new HelloReply
{
Message = "Hello " + request.Name
});
}
}
public class HelloRequest
{
public string Name { get; set; }
}
public class HelloResponse
{
public string Message { get; set; }
}
public class Sender
{
private HttpClient _httpClient;
private HelloRequest _defaultRequest;
public Sender()
{
_httpClient = new HttpClient();
_defaultRequest = new HelloRequest() { Name = "default" };
}
public async Task<string> PostDefault()
{
var content = JsonContent.Create(_defaultRequest);
var response = await _httpClient.PostAsync("http://localhost:5082/greet", content);
var responseString = await response.Content.ReadAsStringAsync();
return JsonSerializer.Deserialize<HelloResponse>(responseString)!.Message;
}
}
var builder = WebApplication.CreateBuilder(args);
// Add services to the container.
var app = builder.Build();
app.MapPost("/greet", (HelloRequest request) =>
{
return new HelloResponse() { Message = "Hello " + request.Name };
});
app.Run();
var summary = BenchmarkDotNet.Running.BenchmarkRunner.Run<BenchmarkWithSamePayload>();
public class BenchmarkWithSamePayload : BenchmarkBase
{
[Benchmark]
public async Task<string> BenchmarkGrpc()
{
return await _gRpcSender.PostDefault();
}
[Benchmark]
public async Task<string> BenchmarkRest()
{
return await _restSender.PostDefault();
}
}
namespace BenchmarkRunner
{
using gRpcSender = GrpcClient.Sender;
using RestSender = RestClient.Sender;
public abstract class BenchmarkBase
{
protected gRpcSender _gRpcSender;
protected RestSender _restSender;
[GlobalSetup]
public void Setup()
{
_gRpcSender = new gRpcSender();
_restSender = new RestSender();
}
}
}
public class BenchmarkConfig : ManualConfig
{
public BenchmarkConfig()
{
AddJob(Job.MediumRun.WithToolchain(InProcessNoEmitToolchain.Instance));
}
}
First of all using microbenchmarking tools for performance testing of web applications can be considered not ideal since web services usually designed to provide better throughput than latency (i.e. handle more load than faster handling a single request).
Secondary (based on the code) you are performing testing in quite specific circumstances - you are testing "simple" responses on the same machine while one of the main goals of gRPC is to reduce size of message send over the network (and arguably better compression might require slower serialization/deserialization) which can be less of the factor in a single threaded test over TCP on the same machine for obvious reasons.
Also the generated GRPC server is using Task's out of the box, so you have a small difference already present there - the REST service does not bother with Task.FromResult which should add some overhead too (though not that much noticable for you REST service).
TL;DR
Use some load testing tool (like NBomber for example) ideally over network with and check the difference in the load REST/GRPC services would be able to handle.