Enable Istio automatic injection only for specified pods within a namespace

We have Istio set up and running in our clusters, with automatic injection enabled by default and enabled in a handful of namespaces. Now we want to do automatic injection for some pods in some other namespaces, but encountered a problem that it is seemingly impossible to do an automatic injection for a specified pod if it is not enabled for the whole namespace. We use Argo workflows to create pods automatically, so we specify sidecar.istio.io/inject: "true" inside Argo workflows so that the resulting pods appear with this annotation in their metadata:

...
metadata:
  annotations:
    sidecar.istio.io/inject: "true"
...

Unfortunately, Istio still does not inject a sidecar unless the namespace has the istio-injection label explicitly set to enabled, adding sidecars to all pods running there.

We cannot use the manual injection either since the pods are created automatically by the Argo service, and we wanted the sidecars to be injected only to specific pods based on the workflow definition.

So are there any possible ways to overcome this issue? Thanks!

Full Argo workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: presto-sql-pipeline-
  annotations: {pipelines.kubeflow.org/kfp_sdk_version: 0.5.1, pipelines.kubeflow.org/pipeline_compilation_time: '2020-05-16T16:07:29.173967',
    pipelines.kubeflow.org/pipeline_spec: '{"description": "Simple demo of Presto
      SQL operator PrestoSQLOp", "name": "Presto SQL Pipeline"}'}
  labels: {pipelines.kubeflow.org/kfp_sdk_version: 0.5.1}
spec:
  entrypoint: presto-sql-pipeline
  templates:
  - name: presto-demo
    container:
      args:
      - --source-name
      - '{{workflow.namespace}}.{{workflow.name}}.presto-demo'
      - --query-sql
      - "SELECT 1;"
      image: gcr.io/our-data-warehouse/presto-cli:latest
      volumeMounts:
      - {mountPath: /mnt/secrets, name: presto-local-vol}
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels: {pipelines.kubeflow.org/pipeline-sdk-type: kfp}
    volumes:
    - name: presto-local-vol
      secret: {secretName: presto-local}
  - name: presto-sql-pipeline
    dag:
      tasks:
      - {name: presto-demo, template: presto-demo}
  arguments:
    parameters: []
  serviceAccountName: argo

Solution

I had a similar requirement - Istio should inject proxy only when specified by the pod and ignore auto injection for all other pods. The solution isnt mentioned in the official docs of Istio but its possible to do so.

As given in this user defined custom matrix, we can have Istio follow this behaviour when the following conditions are met :

The namespace has the label istio-injection=enabled
The Istio global proxy auto inject policy is disabled (For helm chart value : global.proxy.autoInject as given here).
The pod which needs the proxy has the annotation sidecar.istio.io/inject: "true".

All other pods will not have the Istio proxy.