Search code examples
kubernetesdnsamazon-ecsminikube

Building a local testing environment using Minikube and AWS ECS

I am building a local testing environment using Kubernetes Minikube cluster. "Some" of the backend APIs and the database were deployed inside the cluster and each of these APIs has its dedicated URL created using ingress. In addition to that, I've deployed "all" the backend APIS in AWS ECS, each of these APIs has a Route53 record, and the frontend is connected to these APIs in the ".env" file. What I want to achieve is, when I run yarn start on my frontend(React) outside of the cluster, the frontend should first check if the service is presented in the local Minikube cluster, if it can't find the service in the cluster, it will connect to the one in AWS ECS. Is there a way to achieve this?

For better illustration, this is my frontend .env file

SCHEDULE_API_URL = http://schedule.learning.com
DASHBOARD_API_URL = http://dashboard.learning.com

backend deployment yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: schedule
  labels:
    name: schedule
spec:
  replicas: 1
  selector:
    matchLabels:
      app: schedule
  template:
    metadata:
      labels:
        app: schedule
    spec:
      containers:
      - name: schedule
        image: <image_name>
        ports:
        - containerPort: 8080
        imagePullPolicy: Never
      restartPolicy: Always
---
kind: Service
apiVersion: v1
metadata:
  name: schedule-svc
  labels:
    app: schedule
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: schedule-ingress
 namespace: default
 annotations:
   kubernetes.io/ingress.class: "nginx"
   nginx.ingress.kubernetes.io/rewrite-target: /
   nginx.ingress.kubernetes.io/proxy-read-timeout: "12h"
   nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
 rules:
   - host: schedule.learning.com
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: schedule-svc
               port:
                 number: 80

Backend APIs that have been deployed in AWS ECS

SCHEDULE API  --> Route 53 Record: http://schedule.learning.com
DASHBOARD API  --> Route 53 Record: http://dashboard.learning.com

Based on the above example, when I run the frontend locally and outside of the cluster, it will connect to the Schedule API http://schedule.learning.com in the local cluster, but connect to the dashboard API https://dashboard.learning.com in ECS since it couldn't find it in the cluster.

Note:

  • The frontend is deployed outside of the cluster
  • The API in the local cluster will have the same URL as the ECS one so that the frontend env doesn't have to be modified.
  • Even the Minikube cluster is deployed locally, it is still a virtual machine
Solution

  • I think more versatile solution would be if you would have Ingress with two hosts solution. This way you wouldn't need any infrastructure changes (only a few adjustments in names).

    In this solution, you would need to:

    • Enable Ingress in Minikube using $ minikube addons enable ingress
    • Create LoadBalancer to service on cloud environment
    • Have 2 deployments in Cloud environment and Minikube.
    • Add to /etc/hosts aliases for $ minikube ip address like
    192.168.49.2 resource1.domain1.biz
192.168.49.2 resource2.domain2.biz
    • Have 2 services, one which pointing to deployment in Minikube, second should be without selector.
    • Create Endpoint for second service pointing to LoadBalancer IP.

    Configs

    Ingress

    apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress
spec:
  rules:
  - host: resource1.domain1.biz
    http:
      paths:
      - path: /first
        backend:
          serviceName: first-deployment
          servicePort: 8080
  - host: resource2.domain2.biz
    http:
      paths:
      - path: /second
        backend:
          serviceName: my-svc
          servicePort: 8080

    Endpoint

    apiVersion: v1
kind: Endpoints
metadata:
  name: my-svc
subsets:
  - addresses:
      - ip: 34.91.XX.27 
    ports:
      - port: 80

    Tests*

    user@minikube-20:~$ curl -H "HOST: resource1.domain1.biz" http://192.168.49.2/first
Hello, world!
Version: 1.0.0
Hostname: first-deployment-5df6dc8d8b-mmx8c
user@minikube-20:~$ curl -H "HOST: resource2.domain2.biz" http://192.168.49.2/second
Hello, world!
Version: 1.0.0
Hostname: gke-deployment-85b75bf4f9-jgvhz

    In this scenario you could also try to use ExternalName service

    However, if you would like to achieve what you mention originally, which is creating automatic failover you would need to interference your configuration.

    You could use:

    Plugin Alternate is able to selectively forward the query to another upstream server, depending the error result provided by the initial resolver

    • Some kind of LoadBalancer or proxy with health-check and request retry like in this example