Search code examples
amazon-web-serviceskubernetesaws-cloudformationamazon-eksaws-fargate

EKS Fargate pods unreachable from internet


I am trying to create EKS Fargate cluster and deploy example Spring Boot application with 1 endpoint, I successfully create stack with following CloudFormation script:

---
AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS CloudFormation template for EKS Fargate managed Kubernetes cluster with exposed endpoints'

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
      EnableDnsSupport: true
      EnableDnsHostnames: true

  InternetGateway:
    Type: AWS::EC2::InternetGateway

  VPCGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref VPC
      InternetGatewayId: !Ref InternetGateway

  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: 10.0.2.0/24
      MapPublicIpOnLaunch: true
      AvailabilityZone: !Select [ 0, !GetAZs '' ]

  PrivateSubnetA:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: 10.0.0.0/24
      AvailabilityZone: !Select [ 0, !GetAZs '' ]

  PrivateSubnetB:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: 10.0.1.0/24
      AvailabilityZone: !Select [ 1, !GetAZs '' ]

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC

  PublicRoute:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  SubnetRouteTableAssociationA:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet
      RouteTableId: !Ref PublicRouteTable

  EIP:
    Type: AWS::EC2::EIP

  NatGateway:
    Type: AWS::EC2::NatGateway
    Properties:
      SubnetId: !Ref PublicSubnet
      AllocationId: !GetAtt EIP.AllocationId

  PrivateRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC

  PrivateRoute:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGateway

  PrivateSubnetRouteTableAssociationA:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnetA
      RouteTableId: !Ref PrivateRouteTable

  PrivateSubnetRouteTableAssociationB:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnetB
      RouteTableId: !Ref PrivateRouteTable

  EKSCluster:
    Type: AWS::EKS::Cluster
    Properties:
      Name: EKSFargateCluster
      Version: '1.26'
      ResourcesVpcConfig:
        SubnetIds:
          - !Ref PrivateSubnetA
          - !Ref PrivateSubnetB
      RoleArn: !GetAtt EKSClusterRole.Arn

  FargateProfile:
    Type: AWS::EKS::FargateProfile
    Properties:
      ClusterName: !Ref EKSCluster
      FargateProfileName: FargateProfile
      PodExecutionRoleArn: !GetAtt FargatePodExecutionRole.Arn
      Selectors:
        - Namespace: default
      Subnets:
        - !Ref PrivateSubnetA
        - !Ref PrivateSubnetB

  FargateProfileCoredns:
    Type: AWS::EKS::FargateProfile
    Properties:
      ClusterName: !Ref EKSCluster
      FargateProfileName: CorednsProfile
      PodExecutionRoleArn: !GetAtt FargatePodExecutionRole.Arn
      Selectors:
        - Namespace: kube-system
          Labels:
            - Key: k8s-app
              Value: kube-dns
      Subnets:
        - !Ref PrivateSubnetA
        - !Ref PrivateSubnetB

  FargatePodExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - eks-fargate-pods.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonEKSFargatePodExecutionRolePolicy

  EKSClusterRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - eks.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
        - arn:aws:iam::aws:policy/AmazonEKSVPCResourceController

I run following command to path the CoreDNS for Fargate:

kubectl patch deployment coredns \
    -n kube-system \
    --type json \
    -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'

Then I deploy my example application image from public ECR with following kubernetes manifest:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
        - name: ventu
          image: public.ecr.aws/not_real_url/public_ecr_name:latest
          ports:
            - containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: example-service
spec:
  type: LoadBalancer
  selector:
    app: example-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Then when I run:

kubectl get svc

I see result:

NAME              TYPE           CLUSTER-IP      EXTERNAL-IP                                                                  PORT(S)        AGE
example-service   LoadBalancer   172.20.228.77   aa0116829ac2647a7bf39a97bffb0183-1208408433.eu-central-1.elb.amazonaws.com   80:31915/TCP   16m
kubernetes        ClusterIP      172.20.0.1      <none>                                                                       443/TCP        29m

However when I try to reach the EXTERNAL-IP on my LoadBalancer example-service, I get empty response, I can't reach my application on only path defined in my Spring Boot application: /api/v1/info

server.port=8080
server.servlet.context-path=/api/v1

What am I missing?

Couple of information:

  • my pods spin up successfully, I can see Spring Boot logging when I run kubectl logs pod-name
  • my coredns pods spin up correctly as well
  • I use busybox to test my cluster's dns, and everything seems to be working too

Solution

  • I solved my issue, by following this guide

    I then exported resulting stack into my CloudFormation script.

    Then to deploy my application I updated my kubernetes manifest to:

    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      name: example
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      namespace: example
      name: deployment-example-be-app
    spec:
      selector:
        matchLabels:
          app.kubernetes.io/name: example-be-app
      replicas: 2
      template:
        metadata:
          labels:
            app.kubernetes.io/name: example-be-app
        spec:
          containers:
            - name: example-be-app
              image: public.ecr.aws/fake_url/example:latest
              imagePullPolicy: Always
              ports:
                - containerPort: 8080
    ---
    apiVersion: v1
    kind: Service
    metadata:
      namespace: example
      name: service-example-be-app
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        service.beta.kubernetes.io/aws-load-balancer-type: external
        service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    spec:
      type: LoadBalancer
      ports:
        - port: 80
          targetPort: 8080
          protocol: TCP
      selector:
        app.kubernetes.io/name: example-be-app
    

    Now I access my example application form browser.