I'm trying to run a solr server pod on my AKS cluster, but it keeps throwing an error saying - Unable to attach or mount volumes: unmounted volumes=[backup-repository-solr-backup], unattached volumes=[backup-repository-solr-backup kube-api-access-mhz4q solr-xml scgnfs-solr-pvc]: timed out waiting for the condition. I have deployed the cluster, volume mounts and other resources using Terraform.
Some of the kubectl outputs are as shown -
$ kubectl get pods -n <namespace>
NAME READY STATUS RESTARTS AGE
apache-solr-server-solrcloud-0 0/1 Init:0/1 0 3d14h
apache-solr-server-solrcloud-1 0/1 Init:0/1 0 3d14h
apache-solr-server-solrcloud-2 0/1 Init:0/1 0 3d14h
apache-solr-server-solrcloud-zookeeper-0 1/1 Running 0 3d14h
apache-solr-server-solrcloud-zookeeper-1 1/1 Running 1 (3d14h ago) 3d14h
apache-solr-server-solrcloud-zookeeper-2 1/1 Running 4 (25h ago) 3d14h
solr-operator-7c6d46db-qsp4t 1/1 Running 0 3d14h
solr-operator-zookeeper-operator-cfc5d5d59-4hggr 1/1 Running 0 3d14h
$ kubectl describe pod <pod_name> -n <namespace>
Name: apache-solr-server-solrcloud-0
Namespace: cs-solr
Priority: 0
Service Account: default
Node: aks-appsv1-14790217-vmss000010/172.21.145.100
Start Time: Fri, 10 Nov 2023 21:16:32 +0530
Labels: app.kubernetes.io/instance=apache-solr-server
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=solr
app.kubernetes.io/version=8.11.1
controller-revision-hash=apache-solr-server-solrcloud-c9cd4b88c
helm.sh/chart=solr-0.8.0
solr-cloud=apache-solr-server
statefulset.kubernetes.io/pod-name=apache-solr-server-solrcloud-0
technology=solr-cloud
Annotations: solr.apache.org/backupRepositories: solr-backup
solr.apache.org/solrXmlMd5: 28062a041f9071bd95bcf3b0c9ce01b7
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/apache-solr-server-solrcloud
Init Containers:
cp-solr-xml:
Container ID:
Image: library/busybox:1.28.0-glibc
Image ID:
Port: <none>
Host Port: <none>
Command:
sh
-c
cp /tmp/solr.xml /tmp-config/solr.xml && addgroup -g 8983 solr && adduser -u 8983 -G solr -H -D solr && (su solr -c 'test -w /var/solr/data/backup-restore/solr-backup' || chown -R 8983:8983 /var/solr/data/backup-restore/solr-backup)
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 50e-3
memory: 50M
Requests:
cpu: 50e-3
memory: 50M
Environment: <none>
Mounts:
/tmp from solr-xml (rw)
/tmp-config from scgnfs-solr-pvc (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mhz4q (ro)
/var/solr/data/backup-restore/solr-backup from backup-repository-solr-backup (rw,path="cloud/apache-solr-server")
Containers:
solrcloud-node:
Container ID:
Image: solr:8.11
Image ID:
Port: 8983/TCP
Host Port: 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: http-get http://:8983/solr/admin/info/system delay=0s timeout=1s period=20s #success=1 #failure=3
Readiness: http-get http://:8983/solr/admin/info/health delay=0s timeout=1s period=10s #success=1 #failure=2
Startup: http-get http://:8983/solr/admin/info/system delay=10s timeout=1s period=5s #success=1 #failure=10
Environment:
SOLR_JAVA_MEM: -Xms1g -Xmx2g
SOLR_HOME: /var/solr/data
SOLR_PORT: 8983
SOLR_NODE_PORT: 8983
SOLR_PORT_ADVERTISE: 8983
POD_HOSTNAME: apache-solr-server-solrcloud-0 (v1:metadata.name)
POD_NAME: apache-solr-server-solrcloud-0 (v1:metadata.name)
POD_IP: (v1:status.podIP)
POD_NAMESPACE: cs-solr (v1:metadata.namespace)
SOLR_HOST: $(POD_NAME).apache-solr-server-solrcloud-headless.cs-solr
SOLR_LOG_LEVEL: INFO
GC_TUNE:
SOLR_STOP_WAIT: 55
ZK_HOST: apache-solr-server-solrcloud-zookeeper-0.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181,apache-solr-server-solrcloud-zookeeper-1.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181,apache-solr-server-solrcloud-zookeeper-2.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181/
ZK_CHROOT: /
ZK_SERVER: apache-solr-server-solrcloud-zookeeper-0.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181,apache-solr-server-solrcloud-zookeeper-1.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181,apache-solr-server-solrcloud-zookeeper-2.apache-solr-server-solrcloud-zookeeper-headless.cs-solr.svc.cluster.local:2181
SOLR_OPTS: -DhostPort=$(SOLR_NODE_PORT)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mhz4q (ro)
/var/solr/data from scgnfs-solr-pvc (rw)
/var/solr/data/backup-restore/solr-backup from backup-repository-solr-backup (rw,path="cloud/apache-solr-server")
Readiness Gates:
Type Status
solr.apache.org/isNotStopped True
Conditions:
Type Status
solr.apache.org/isNotStopped True
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
scgnfs-solr-pvc:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: scgnfs-solr-pvc-apache-solr-server-solrcloud-0
ReadOnly: false
solr-xml:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: apache-solr-server-solrcloud-configmap
Optional: false
backup-repository-solr-backup:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: scgnfs-solr-pvc
ReadOnly: false
kube-api-access-mhz4q:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 19m (x979 over 3d13h) kubelet Unable to attach or mount volumes: unmounted volumes=[backup-repository-solr-backup], unattached volumes=[solr-xml scgnfs-solr-pvc backup-repository-solr-backup kube-api-access-mhz4q]: timed out waiting for the condition
Warning FailedMount 10m (x437 over 3d14h) kubelet Unable to attach or mount volumes: unmounted volumes=[backup-repository-solr-backup], unattached volumes=[scgnfs-solr-pvc backup-repository-solr-backup kube-api-access-mhz4q solr-xml]: timed out waiting for the condition
Warning FailedMount 3m29s (x431 over 3d13h) kubelet Unable to attach or mount volumes: unmounted volumes=[backup-repository-solr-backup], unattached volumes=[backup-repository-solr-backup kube-api-access-mhz4q solr-xml scgnfs-solr-pvc]: timed out waiting for the condition
$ kubectl get pvc -n <namespace>
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-apache-solr-server-solrcloud-zookeeper-0 Bound pvc-a520af21-4084-4af5-b954-4613697a43ac 20Gi RWO default 3d14h
data-apache-solr-server-solrcloud-zookeeper-1 Bound pvc-3f27a00f-6577-4126-a014-f13c5ab4b881 20Gi RWO default 3d14h
data-apache-solr-server-solrcloud-zookeeper-2 Bound pvc-5e9a1e38-7f74-4270-9ac6-02df453190e4 20Gi RWO default 3d14h
scgnfs-solr-pvc Bound scgnfs-solr-pv 100 RWX azurefile-csi-premium 3d15h
scgnfs-solr-pvc-apache-solr-server-solrcloud-0 Bound pvc-ba419a5f-f634-4aa5-97f4-01a6066bb227 100Gi RWO azurefile-csi-premium 3d14h
scgnfs-solr-pvc-apache-solr-server-solrcloud-1 Bound pvc-2acf3702-3e38-4c89-9366-7cd8336aee88 100Gi RWO azurefile-csi-premium 3d14h
scgnfs-solr-pvc-apache-solr-server-solrcloud-2 Bound pvc-c13b240b-e133-46a5-b2ba-50f12d0aea07 100Gi RWO azurefile-csi-premium 3d14h
$ kubectl describe pvc scgnfs-solr-pvc -n <namespace>
Name: scgnfs-solr-pvc
Namespace: cs-solr
StorageClass: azurefile-csi-premium
Status: Bound
Volume: scgnfs-solr-pv
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 100
Access Modes: RWX
VolumeMode: Filesystem
Used By: apache-solr-server-solrcloud-0
apache-solr-server-solrcloud-1
apache-solr-server-solrcloud-2
Events: <none>
The values.yaml file passed while configuring the solr-server helm chart -
version: 0.8.0
backupRepositories:
- name: "solr-backup"
volume:
source:
persistentVolumeClaim:
claimName: "scgnfs-solr-pvc"
dataStorage:
type: persistent
capacity: 100Gi
persistent:
pvc:
name: scgnfs-solr-pvc
storageClassName: azurefile-csi-premium
resources:
limits:
cpu: 4000m
memory: 4G
The PVC for the backup repository is same as the PVC for solr-server. Is that an issue? I don't know where the issue is. The PVs and PVCs are all seemed to be successfully applied.
Can anyone help me where I'm wrong exactly? Thanks in advance.
Edit1: I created a different PV/PVC for backupRepository but that also gave the same error.
So I was able to figure out the reason for the error. It turns out that the solr backup repository requires only a PVC and not the corresponding PV. It creates its own volume. I was earlier creating a PV and mapping this "scgnfs-solr-pvc" to that PV and then passing this pvc for the backup repository which is why the pvc was unable to attach to the volume solr was creating for the backup repository. You don't have to map this pvc with any pv. Create a standalone pvc without any volume and attach it to the backup repository and the pods will start running.