GRPC service inside Kubernetes is working but fails with an GRPC protocol error when we use istio

Question

I have a server to server calls and I use GRPC (with .net core 5) It's working and test in local.

After that, I have moved all the services to Kubernetes Pod (Docker Desktop) and also tested through the flow (with swagger post-call) and it's working there too.

Now for monitoring, I added ISTIO and added the label to my namespace "istio-injection=enabled" restarted all my pods and now all are having 2 containers inside each pod.

I tested the basic services (again swagger) and it's working. when it comes to testing the GRPC call. The call is failing from the caller side saying

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="upstream connect error or disconnect/reset before headers. reset reason: protocol error")

I checked the logs at GRPC server-side and it has no clue about this call and the service is just running. then I am kind of thinking that error is coming from the caller side whereas it is not able to or make a call to GRPC server.

The error detail:

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="upstream connect error or disconnect/reset before headers. reset reason: protocol error") 
at Basket.API.GrpcServices.DiscountGrpcService.GetDiscount(String productName) in /src/Services/Basket/Basket.API/GrpcServices/DiscountGrpcService.cs:line 21 
at Basket.API.Controllers.BasketController.UpdateBasket(ShoppingCart basket) in /src/Services/Basket/Basket.API/Controllers/BasketController.cs:line 47 at lambda_method7(Closure , Object ) 
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Obje

Again, I remove the Istio and tested and that's started working again (without changing anything) I added istio back and it started failing again. all other services are working with istio but not this call (This is only GRPC call I have).

Answer 1

I found a solution at https://istiobyexample.dev/grpc/ where it describes the missing item.

istio recommends using the name and version tag as Label but more importantly when working with GRPC, the service that exposes the GRPC needs to have the port name GRPC.

I have added that restarted the service and it got started working as expected. Again it's not something I resolved. All credit goes to the link https://istiobyexample.dev/grpc/ and the image posted below.