Kubernetes NiFi Cluster setup in AKS

Question

Thought to post this because it might help someone. I couldn't find Kubernetes NiFi setup without helm package, so I have prepared the below configuration YAML for Kubernetes NiFi Cluster setup.

Here's the link for Zookeeper Cluster setup in AKS

Answer 1

Please comment if you see any issues anywhere in the configuration or if you would like to provide any suggestions. Increase disk storage configuration according to your usage.

apiVersion: v1
kind: Service
metadata:
  name: nifi-hs
  labels:
    app: nifi
spec:
  ports:
  - port: 1025
    name: nodeport
  - port: 8080
    name: client
  clusterIP: None
  selector:
    app: nifi
---
apiVersion: v1
kind: Service
metadata:
  name: nifi-cs
  labels:
    app: nifi
  annotations:
      service.beta.kubernetes.io/azure-dns-label-name: nifi
spec:
  ports:
  - port: 80
    targetPort: 8080
    name: client
  selector:
    app: nifi
  type: LoadBalancer
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: nifi-pdb
spec:
  selector:
    matchLabels:
      app: nifi
  maxUnavailable: 1
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nifi-sc
  selfLink: /apis/storage.k8s.io/v1/storageclasses/nifi-sc
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: 'true'
provisioner: kubernetes.io/azure-disk
parameters:
  cachingmode: ReadOnly
  kind: Managed
  storageaccounttype: StandardSSD_LRS
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nifi
spec:
  selector:
    matchLabels:
      app: nifi
  serviceName: nifi-hs
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: OrderedReady
  template:
    metadata:
      labels:
        app: nifi
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - k: "app"
                    operator: In
                    values:
                    - nifi
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: nifi        
        image: "apache/nifi:1.13.0"
        env:           
        - name: NIFI_CLUSTER_IS_NODE
          value: "true"
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: NIFI_CLUSTER_ADDRESS
          value: $(HOSTNAME).nifi-hs
        - name: NIFI_CLUSTER_NODE_PROTOCOL_PORT
          value: "1025"
        - name: NIFI_WEB_HTTP_HOST
          value: $(HOSTNAME).nifi-hs.ns1.svc.cluster.local
        #- name: NIFI_WEB_HTTP_PORT
        #  value: "80"
        - name: NIFI_CLUSTER_NODE_PROTOCOL_MAX_THREADS
          value: "100" 
        - name: NIFI_ZK_CONNECT_STRING
          value: "zk-cs:2181" 
        - name: NIFI_ELECTION_MAX_CANDIDATES
          value: "3" 
        ports:
        - containerPort: 8080
          name: client
        - containerPort: 1025
          name: nodeport
        volumeMounts:        
        - name: nifi-database
          mountPath: "/opt/nifi/nifi-current/database_repository"
        - name: nifi-flowfile
          mountPath: "/opt/nifi/nifi-current/flowfile_repository"
        - name: nifi-content
          mountPath: "/opt/nifi/nifi-current/content_repository"
        - name: nifi-provenance
          mountPath: "/opt/nifi/nifi-current/provenance_repository"
        - name: nifi-state
          mountPath: "/opt/nifi/nifi-current/state"
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: nifi-database
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-flowfile
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-content
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-provenance
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-state
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi