Search code examples
dockerkubernetesdocker-composekompose

Converting docker-compose using Kompose to deploy workloads on GKE

I have project written in Django Restframework, Celery for executing long running task, Redis as a broker and Flower for monitoring Celery task. I have written a Dockerfile & docker-compose.yaml to create a network and run this services inside containers.

Dockerfile

FROM python:3.7-slim

ENV PYTHONUNBUFFERED 1

RUN apt-get update &&\
    apt-get install python3-dev default-libmysqlclient-dev gcc  -y &&\
    apt-get install -y libssl-dev libffi-dev &&\
    python -m pip install --upgrade pip &&\
    mkdir /ibdax

WORKDIR /ibdax

COPY ./requirements.txt /requirements.txt

COPY . /ibdax

EXPOSE 80

EXPOSE 5555

ENV ENVIRONMENT=LOCAL

#install dependencies
RUN pip install -r /requirements.txt

RUN pip install django-phonenumber-field[phonenumbers]
RUN pip install drf-yasg[validation]

docker-compose.yaml

version: "3"

services:
  redis:
    container_name: redis-service
    image: "redis:latest"
    ports:
      - "6379:6379"
    restart: always
    command: "redis-server"

  ibdax-backend:
    container_name: ibdax
    build:
      context: .
      dockerfile: Dockerfile
    image: "ibdax-django-service"
    volumes:
      - .:/ibdax
    ports:
      - "80:80"
    expose:
      - "80"
    restart: always
    env_file:
      - .env.staging
    command: >
      sh -c "daphne -b 0.0.0.0 -p 80 ibdax.asgi:application"
    links:
      - redis

  celery:
    container_name: celery-container
    image: "ibdax-django-service"
    command: "watchmedo auto-restart -d . -p '*.py' -- celery  -A ibdax worker -l INFO"
    volumes:
      - .:/ibdax
    restart: always
    env_file:
      - .env.staging
    links:
      - redis
    depends_on:
      - ibdax-backend

  flower:
    container_name: flower
    image: "ibdax-django-service"
    command: "flower -A ibdax --port=5555 --basic_auth=${FLOWER_USERNAME}:${FLOWER_PASSWORD}"
    volumes:
      - .:/ibdax
    ports:
      - "5555:5555"
    expose:
      - "5555"
    restart: always
    env_file:
      - .env
      - .env.staging
    links:
      - redis
    depends_on:
      - ibdax-backend

This Dockerfile & docker-compose is working just fine and now I want to deploy this application to GKE. I came across Kompose which translate the docker-compose to kubernetes resources. I read the documentation and started following the steps and the first step was to run kompose convert. This returned few warnings and created few files as show below -

WARN Service "celery" won't be created because 'ports' is not specified 
WARN Volume mount on the host "/Users/jeetpatel/Desktop/projects/ibdax" isn't supported - ignoring path on the host 
WARN Volume mount on the host "/Users/jeetpatel/Desktop/projects/ibdax" isn't supported - ignoring path on the host 
WARN Volume mount on the host "/Users/jeetpatel/Desktop/projects/ibdax" isn't supported - ignoring path on the host 
INFO Kubernetes file "flower-service.yaml" created 
INFO Kubernetes file "ibdax-backend-service.yaml" created 
INFO Kubernetes file "redis-service.yaml" created 
INFO Kubernetes file "celery-deployment.yaml" created 
INFO Kubernetes file "env-dev-configmap.yaml" created 
INFO Kubernetes file "celery-claim0-persistentvolumeclaim.yaml" created 
INFO Kubernetes file "flower-deployment.yaml" created 
INFO Kubernetes file "flower-claim0-persistentvolumeclaim.yaml" created 
INFO Kubernetes file "ibdax-backend-deployment.yaml" created 
INFO Kubernetes file "ibdax-backend-claim0-persistentvolumeclaim.yaml" created 
INFO Kubernetes file "redis-deployment.yaml" created

I ignored the warnings and moved to the next step i.e running command

kubectl apply -f flower-service.yaml, ibdax-backend-service.yaml, redis-service.yaml, celery-deployment.yaml

but I get this error -

error: Unexpected args: [ibdax-backend-service.yaml, redis-service.yaml, celery-deployment.yaml]

Hence I planned to apply one by one like this -

kubectl apply -f flower-service.yaml

but I get this error -

The Service "flower" is invalid: spec.ports[1]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:5555, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}

Not sure where am I going wrong.

Also the prerequisites of Kompose is to have a Kubernetes cluster so I created an Autopilot cluster with public network. Now I am not sure how this apply command will identify the cluster I created and deploy my application on it.

Solution

  • After kompose convert your flower-service.yaml file have duplicate ports - that's what the error is saying.

    ...
  ports:
    - name: "5555"
      port: 5555
      targetPort: 5555
    - name: 5555-tcp
      port: 5555
      targetPort: 5555
...

    You can either delete port name: "5555" or name: 5555-tcp.
    For example, replace ports block with

      ports:
    - name: 5555-tcp
      port: 5555
      targetPort: 5555

    and deploy the service again.
    I would also recommend changing port name to something more descriptive.

    Same thing happens with ibdax-backend-service.yaml file.

    ...
  ports:
    - name: "80"
      port: 80
      targetPort: 80
    - name: 80-tcp
      port: 80
      targetPort: 80
...

    You can delete one of the definitions, and redeploy the service (changing port name to something more descriptive is also recommended).

    kompose is not a perfect tool, that will always give you a perfect result. You should check the generated files for any possible conflicts and/or missing fields.