If I have a GKE cluster with a few GRPC microservices that are only exposed through ClusterIP services, is it necessary to use mTLS/Istio? If the services are only talking to each other on the same VPC, it seems like overkill, but I hate to see "withInsecure" all over the codebase. We're terminating TLS using a GKE Ingress for our front end, but it seems like unsecured VPC-internal traffic should be fine.
Although it is not necessary, it is definitely recommended to use them. This way you have encryption inside your mesh which is appreciated on production environments and if you are using Istio you will have deep insights and observability over your microservices, as Istio has Prometheus, Grafana and Kiali as auditing tools that will ease the management of the mesh.
If you are planning on deploying Istio you should definitely check out their Getting Started guide. GKE actually has an Istio plugin that you can check here that installs Istio 1.2.10 and you only need to set up the configuration for your application.