Do I need to create different StorageClasses for different PVs?

Question

I have a NAS (TrueNAS/FreeNAS) through which I want to provide Persistent Volumes to my Kubernetes cluster.

I am sharing the directories using SMB share. The share will have different users with different permissions (mainly readOnly vs read and write).

I will then be creating multiple Persistent Volumes (and PVCs as they are one-to-one) with different access modes and corresponding credentials. This is to make sure that services have limited access to the directories as per their usage.

Currently, SMB is something that is providing me with good performance as well as let's me configure access to its different directories through setting up users and their ACLs.

Currently, I am using K3S for setting up the Kubernetes cluster.

I looked online and found two csi driver implementations that will help me do that i.e. csi-driver-smb and democratic-csi.

However, in both of their configurations of StorageClass this and that, both ask for username and passwords and the smb share endpoint in the storage class itself. Are storage classes suppose to contain the credentials of the storage itself? I was assuming that it would define how to connect to a SMB storage and PV would provide necessary credentials to do that. It also seems to be asking for the smb share endpoint in the storage class.

Is my definition of storage classes wrong?

For my use case, where I need configure two accesses with different permissions to the same smb directory, do I need to create two storage classes?

Also, if I need to connect to another directory shared via SMB, do I again need to create a new storage class?

Answer 1

YES. You absolutely should create different storage classes for different PVs that require different configurations. Think groups/sets of PVs though, not per each PV. I use many CSIs including the mentioned csi-driver-smb.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: smb-q2-default
  labels:
    app.kubernetes.io/managed-by: Helm
  annotations:
    meta.helm.sh/release-name: csi-driver-smb-q2
    meta.helm.sh/release-namespace: csi-driver-smb-q2
provisioner: smbq2.csi.k8s.io
reclaimPolicy: Retain
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - vers=3.0
allowVolumeExpansion: false
volumeBindingMode: Immediate

Can you group your PVs into sets with common retention, expansion, and mount options ? Make a storage class for each of them.

You asked another question about where credentials are defined - this is CSI dependent but typically pretty flexible. csi-driver-smb for example, you could set credentials per PV, like so:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example
spec:
  capacity:
    storage: 1Gi
  csi:
    driver: smbq2.csi.k8s.io
    volumeHandle: globally-unique-id
    fsType: ext4
    volumeAttributes:
      createSubDir: 'true'
      source: \\unc.server.path\config\example
    nodeStageSecretRef:
      name: example-smb-secret
      namespace: csi-smb-q2
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: smb-default
  volumeMode: Filesystem

The format of the SMB credentials / secret can vary based on OS, domain, etc. you'll figure it out.

Also, it's worth mentioning - in recent k8s versions - if you have an existing PV that doesn't reference a storageClass, it can be patched in-place, one time, without recreation.