When using kubernetes(minikube) statefulset on local machine, EMQX Rules are persisting because same pod IP is being assigned to the emqx node, for example /opt/emqx/data/mnesia/[email protected]. Even if I delete the pod when the new pod starts, it gets assigned the same IP as before. Everything is working as it should.
But when I'm using aks(azure kubernetes) to deploy EMQX on aks cluster using azure files, pod IP is different everytime. For example if /opt/emqx/data/mnesia/[email protected] is assigned to the EMQX node, then if I try to delete the pod then /opt/emqx/data/mnesia/[email protected] might be assigned to it.
so, nothing is persisting.
Local code
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage5
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: emqx-pv5
spec:
capacity:
storage: 300Mi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage5
local:
path: /opt/emqx/data/mnesia
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- minikube
---
apiVersion: v1
kind: Service
metadata:
name: emqx-headless
spec:
type: ClusterIP
clusterIP: None
selector:
app: emqx
ports:
- name: mqtt
port: 1883
protocol: TCP
targetPort: 1883
- name: mqttssl
port: 8883
protocol: TCP
targetPort: 8883
- name: mgmt
port: 8081
protocol: TCP
targetPort: 8081
- name: websocket
port: 8083
protocol: TCP
targetPort: 8083
- name: wss
port: 8084
protocol: TCP
targetPort: 8084
- name: dashboard
port: 18083
protocol: TCP
targetPort: 18083
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: emqx-statefulset
labels:
app: emqx
spec:
replicas: 1
serviceName: emqx-headless
selector:
matchLabels:
app: emqx
template:
metadata:
labels:
app: emqx
spec:
containers:
- name: emqx
image: emqx/emqx:4.2.7
ports:
- name: emqx-dashboard
containerPort: 18083
- name: ssl-port
containerPort: 8883
- name: emqx-port
containerPort: 1883
- name: ssl-dashboard
containerPort: 18084
env:
- name: EMQX_LOADED_PLUGINS
value: emqx_management,emqx_recon,emqx_retainer,emqx_dashboard,emqx_rule_engine,emqx_auth_username
- name: EMQX_CLUSTER__DISCOVERY
value: k8s
- name: EMQX_NAME
value: emqx
- name: EMQX_CLUSTER__K8S__APISERVER
value: https://kubernetes.default:443
- name: EMQX_CLUSTER__K8S__SERVICE_NAME
value: emqx
- name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
value: ip
- name: EMQX_CLUSTER__K8S__APP_NAME
value: emqx
- name: EMQX_ALLOW_ANONYMOUS
value: "false"
- name: EMQX_LISTENER__SSL__EXTERNAL__MAX_CONNECTIONS
value: "1024000"
- name: EMQX_AUTH__USER__PASSWORD_HASH
value: sha256
- name: EMQX_AUTH__USER__1__USERNAME
value:
- name: EMQX_AUTH__USER__1__PASSWORD
value:
- name: EMQX_DASHBOARD__DEFAULT_USER__LOGIN
value:
- name: EMQX_DASHBOARD__DEFAULT_USER__PASSWORD
value:
- name: EMQX_DASHBOARD__LISTENER__HTTPS
value: "18084"
- name: MQX_DASHBOARD__LISTENER__HTTPS__ACCEPTORS
value: "4"
- name: EMQX_DASHBOARD__LISTENER__HTTPS__MAX_CLIENTS
value: "512"
tty: true
volumeMounts:
- name: emqx-mnesia
mountPath: "/opt/emqx/data/mnesia"
volumeClaimTemplates:
- metadata:
name: emqx-mnesia
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "local-storage5"
resources:
requests:
storage: 300Mi
Azure Kubernetes code
apiVersion: v1
kind: ServiceAccount
metadata:
name: emqx
namespace: emqx-test
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: emqx
subjects:
- kind: ServiceAccount
name: emqx
namespace: emqx-test
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: emqx-files
provisioner: kubernetes.io/azure-file
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=0
- gid=0
- mfsymlinks
- cache=strict
- actimeo=30
parameters:
skuName: Standard_LRS
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: emqx-pvc
namespace: emqx-test
spec:
accessModes:
- ReadWriteMany
storageClassName: emqx-files
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: emqx
namespace: emqx-test
spec:
ports:
- name: emqx-dashboard
port: 80
targetPort: 18083
protocol: TCP
- name: ssl-port
port: 8883
targetPort: ssl-port
protocol: TCP
- name: emqx-port
port: 1883
targetPort: emqx-port
protocol: TCP
- name: ssl-dashboard
port: 443
targetPort: 18084
protocol: TCP
selector:
app: emqx
type: LoadBalancer
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: emqx
labels:
app: emqx
namespace: emqx-test
spec:
serviceName: "emqx"
selector:
matchLabels:
app: emqx
replicas: 1
template:
metadata:
labels:
app: emqx
spec:
containers:
- name: emqx
image: emqx/emqx:4.2.7
ports:
- name: emqx-dashboard
containerPort: 18083
- name: ssl-port
containerPort: 8883
- name: emqx-port
containerPort: 1883
- name: ssl-dashboard
containerPort: 18084
env:
- name: EMQX_LOADED_PLUGINS
value: emqx_management,emqx_recon,emqx_retainer,emqx_dashboard,emqx_rule_engine,emqx_auth_username
- name: EMQX_CLUSTER__DISCOVERY
value: k8s
- name: EMQX_NAME
value: emqx
- name: EMQX_CLUSTER__K8S__APISERVER
value: https://kubernetes.default:443
- name: EMQX_CLUSTER__K8S__NAMESPACE
value: emqx-test
- name: EMQX_CLUSTER__K8S__SERVICE_NAME
value: emqx
- name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
value: ip
- name: EMQX_CLUSTER__K8S__APP_NAME
value: emqx
- name: EMQX_ALLOW_ANONYMOUS
value: "false"
- name: EMQX_LISTENER__SSL__EXTERNAL__MAX_CONNECTIONS
value: "1024000"
- name: EMQX_AUTH__USER__PASSWORD_HASH
value: sha256
- name: EMQX_AUTH__USER__1__USERNAME
value:
- name: EMQX_AUTH__USER__1__PASSWORD
value:
- name: EMQX_DASHBOARD__DEFAULT_USER__LOGIN
value:
- name: EMQX_DASHBOARD__DEFAULT_USER__PASSWORD
value:
- name: EMQX_DASHBOARD__LISTENER__HTTPS
value: "18084"
- name: MQX_DASHBOARD__LISTENER__HTTPS__ACCEPTORS
value: "4"
- name: EMQX_DASHBOARD__LISTENER__HTTPS__MAX_CLIENTS
value: "512"
volumeMounts:
- name: emqx-data
mountPath: "/opt/emqx/data/mnesia"
tty: true
volumes:
- name: emqx-data
persistentVolumeClaim:
claimName: emqx-pvc
In k8s documentation on StatefulSet Basics you read:
The Pods' ordinals, hostnames, SRV records, and A record names have not changed, but the IP addresses associated with the Pods may have changed. In the cluster used for this tutorial, they have. This is why it is important not to configure other applications to connect to Pods in a StatefulSet by IP address.
This is expected and as you see and this behaviour is mentioned in documentation.
But why do you see different behavior on minikube and different on azure? IP addresses are assigned by CNIs. On minikube default CNI it the docker-bridge, and on azure its's Azure CNI, so it is up to the CNI what address is assigned.
It's best to always assume that you cannot rely on pod IP addresses to stay static. Use DNS for statefulsets and for other pods and services for communication and never use hardcoded pod ip addresses directly.