Kubernetes service to load balance multiple pods

Perhaps I'm not searching correctly or I'm not understanding a fundamental piece of the puzzle.

I have an application which is an ASP.NET web api serving http requests. It will have let's say 3 replicas. In addition to the three replicas, I want a 4th replica (or instance) of the exact same image, but an environment variable changed. This fourth instance will be using all the same code and processing background requests from an AWS Kinesis stream.

To prevent handling the messages more than once, we want the 4th instance to be the only instance handling the kinesis messages.

We're using helm.

I've got this part working pretty well with 2 separate deployments, one has 3 replicas, and the other has 1, easy.

The part I cannot figure out how to make the 4th instance ALSO eligible to serve HTTP requests. Since the kinesis messages are low volume, there's little to no concern with using the resources for the listener pod to also serve http requests.

How can I make a kubernetes service that can load balance between the two deployments?

I've tried something like this

service.yaml

spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
    - port: 80
      targetPort: http-observer
      protocol: TCP
      name: http-observer

deployment-api.yaml

ports:
  - name: http
    containerPort: 8800
    protocol: TCP

deployment-observer.yaml

ports:
  - name: http-observer
    containerPort: 8800
    protocol: TCP

This yields the following error though:

 Error: Service "my-service-name-here" is invalid: spec.ports[1]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:80, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}

Presumeably because I can't have 2 port 80s on the same service.

I want a service exposing port 80 to equally load balance according to whatever k8s HTTP traffic load balancing rules exist naturally between port 8800 on the three api pods and port 8800 on the observer pod. Is this possible? Am I going about this all wrong?

Any guidance would be appreciated.

NOTE: Due to build pipeline, and dependency management, it's important that the two pods remain one code base, so we're not entertaining splitting the kinesis listener code and the API code into two separate apps with a shared library.

Solution

What you're trying to achieve is to basically split traffic between different versions of the same application.

The easiest option is to use kubernetes internals for that. You can use one service that serves both deployments by setting the same label/selector on all three.

kind: Service
[...]
spec:
  selector:
    app.kubernetes.io/name: my-service
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http

# Deployment 1
kind: Deployment
[...]
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: my-service
  template:
    metadata:
      labels:
        app.kubernetes.io/name: my-service
---
# Deployment 2
kind: Deployment
[...]
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: my-service
  template:
    metadata:
      labels:
        app.kubernetes.io/name: my-service

By that kubernetes will split traffic between those 4 pods, where the second deployment is the one with a different environment variable.

If you need more sophisticated traffic splitting you could use something like istio to setup different versions and let istio handle the split, see docs (it's for canary deployments, but you could use it for your use case as well).