Search code examples
kubernetesgoogle-kubernetes-engineobject-storagefile-storageblock-storage

How kubernetes block, file and object storage types are used inside containers

In the context of Kubernetes, I've come across the terms Block Storage, File Storage and Object Storage but I don't understand how they are really used (mounted) inside a container. I have a few questions,

  1. Are these all storage types backed by raw block devices?
  2. Is Block Storage a term used to mean a logical abstraction of block devices?
  3. Is Block Storage mounted to a path inside a container just like we mount a file system on linux? which also implies the question whether the Block Storage is a formatted file system?
  4. How Object Storage is presented to a container? How does the container make use of it? Is it mounted to a path?
  5. How File Storage is presented to a container? How does the container make use of it? Is it mounted to a path?
  6. What are 3 example scenarios to use these 3 storage types?
Solution

  • Block storage is backed by block device. It can be physical disk or it can be network-attached device (iSCSI, FC or AWS EBS volume) or even Ceph RBD. In most cases pods don't need to work with raw block devices (with exception of Kube native storages like Ceph, Portworx) and Kubernetes instead creates filesystem on top of it and mounts it into pod. The main thing about block storage is that in most cases it's Read-Write Only (RWO) which means it can be mounted read-write only to single pod.

    File storage is backed by filesystem. It can be local filesystem, like hostPath, or it can be network share like NFS. In that case Kubernetes can directly mount it inside pod without any additional preparation. The main thing about NFS is that it can be mounted Read-Write Many (RWX) which means it can be mounted read-write to many pods. Also filesystems on one node can be attached to many pods on that particular node.

    Object storage can be imagined like files-over-HTTP(S) (AWS S3, GCP GCS, Azure Blob Storage, Ceph RGW, Minio). There is no official Kubernetes supported way to mount object storage inside pods, but there are some dirty workarounds like s3fs, Ganesha NFS and may be others. In most cases you will work with object storage directly from your app using provider specific libraries which is how it's meant to work.