consul StatefulSet failing

Question

I am trying to deploy consul using kubernetes StatefulSet with following manifest

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: consul
  labels:
    app: consul
rules:
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
      - list

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: consul
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: consul
subjects:
  - kind: ServiceAccount
    name: consul
    namespace: dev-ethernet

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: consul
  namespace: dev-ethernet
  labels:
    app: consul

---

apiVersion: v1
kind: Secret
metadata:
  name: consul-secret
  namespace: dev-ethernet
data:
  consul-gossip-encryption-key: "aIRpNkHT/8Tkvf757sj2m5AcRlorWNgzcLI4yLEMx7M="

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: consul-config
  namespace: dev-ethernet
data:
  server.json: |
    {
      "bind_addr": "0.0.0.0",
      "client_addr": "0.0.0.0",
      "disable_host_node_id": true,
      "data_dir": "/consul/data",
      "log_level": "INFO",
      "datacenter": "us-west-2",
      "domain": "cluster.local",
      "ports": {
        "http": 8500
      },
      "retry_join": [
        "provider=k8s label_selector=\"app=consul,component=server\""
      ],
      "server": true,
      "telemetry": {
        "prometheus_retention_time": "5m"
      },
      "ui": true
    }

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: consul
  namespace: dev-ethernet
spec:
  selector:
    matchLabels:
      app: consul
      component: server
  serviceName: consul
  podManagementPolicy: Parallel
  replicas: 3
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: consul
        component: server
      annotations:
        consul.hashicorp.com/connect-inject: "false"
    spec:
      serviceAccountName: consul
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - consul
              topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 10
      securityContext:
        fsGroup: 1000
      containers:
        - name: consul
          image: "consul:1.8"
          args:
            - "agent"
            - "-advertise=$(POD_IP)"
            - "-bootstrap-expect=3"
            - "-config-file=/etc/consul/config/server.json"
            - "-encrypt=$(GOSSIP_ENCRYPTION_KEY)"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: GOSSIP_ENCRYPTION_KEY
              valueFrom:
                secretKeyRef:
                  name: consul-secret
                  key: consul-gossip-encryption-key
          volumeMounts:
            - name: data
              mountPath: /consul/data
            - name: config
              mountPath: /etc/consul/config
          lifecycle:
            preStop:
              exec:
                command:
                - /bin/sh
                - -c
                - consul leave
          ports:
            - containerPort: 8500
              name: ui-port
            - containerPort: 8400
              name: alt-port
            - containerPort: 53
              name: udp-port
            - containerPort: 8080
              name: http-port
            - containerPort: 8301
              name: serflan
            - containerPort: 8302
              name: serfwan
            - containerPort: 8600
              name: consuldns
            - containerPort: 8300
              name: server
      volumes:
        - name: config
          configMap:
            name: consul-config
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: consul
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: aws-gp2
      resources:
        requests:
          storage: 3Gi

But gets ==> encrypt has invalid key: illegal base64 data at input byte 1 when container starts.

I have generated consul-gossip-encryption-key locally using docker run -i -t consul keygen

Anyone knows whats wrong here ?

Answer 1

secret.data must be base64 string.

try

kubectl create secret generic consul-gossip-encryption-key --from-literal=key="$(docker run -i -t consul keygen)" --dry-run -o=yaml

and replace

apiVersion: v1
kind: Secret
metadata:
  name: consul-secret
  namespace: dev-ethernet
data:
  consul-gossip-encryption-key: "aIRpNkHT/8Tkvf757sj2m5AcRlorWNgzcLI4yLEMx7M="

ref: https://www.consul.io/docs/k8s/helm#v-global-gossipencryption