Search code examples
kuberneteshashicorp-vault

HashiCorp Vault Enable REST Call

I am following the HashiCorp tutorial and it all looks fine until I try to launch the "webapp" pod - a simple pod whose only function is to demonstrate that it can start and mount a secret volume.

The error (permission denied on a REST call) is shown at the bottom of this command output:

kubectl describe pod webapp
Name:             webapp
Namespace:        default
Priority:         0
Service Account:  webapp-sa
Node:             docker-desktop/192.168.65.4
Start Time:       Tue, 14 Feb 2023 09:32:07 -0500
Labels:           <none>
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Containers:
  webapp:
    Container ID:
    Image:          jweissig/app:0.0.1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/secrets-store from secrets-store-inline (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5b76r (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=vault-database
  kube-api-access-5b76r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    42m                 default-scheduler  Successfully assigned default/webapp to docker-desktop
  Warning  FailedMount  20m (x8 over 40m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline kube-api-access-5b76r]: timed out waiting for the condition
  Warning  FailedMount  12m (x23 over 42m)  kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/webapp, err: rpc error: code = Unknown desc = error making mount request: couldn't read secret "db-password": Error making API request.

URL: GET http://vault.default:8200/v1/secret/data/db-pass
Code: 403. Errors:

* 1 error occurred:
           * permission denied
  Warning  FailedMount  2m19s (x4 over 38m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[kube-api-access-5b76r secrets-store-inline]: timed out waiting for the condition

So it seems that this REST call fails: GET http://vault.default:8200/v1/secret/data/db-pass. Indeed, it fails from curl as well:

curl -vik -H "X-Vault-Token: root"  http://localhost:8200/v1/secret/data/db-pass
*   Trying 127.0.0.1:8200...
* TCP_NODELAY set
* connect to 127.0.0.1 port 8200 failed: Connection refused
* Failed to connect to localhost port 8200: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 8200: Connection refused

At this point I am a bit lost. I am not sure that the REST call is configured correctly, i.e. in such a way that Vault will accept it; but I am also not sure how to configure it differently.

The Vault logs show the information below, so I seems that the port and token I use are correct:

2023-02-14 09:07:14 You may need to set the following environment variables:
2023-02-14 09:07:14     $ export VAULT_ADDR='http://[::]:8200'
2023-02-14 09:07:14 The root token is displayed below
2023-02-14 09:07:14 Root Token: root

Vault seems to be running fine in Kubernetes:

kubectl get pods
NAME                                    READY   STATUS    RESTARTS      AGE
vault-0                                 1/1     Running   1 (22m ago)   32m
vault-agent-injector-77fd4cb69f-mf66p   1/1     Running   1 (22m ago)   32m

If I try to show the Vault status:

vault status
Error checking seal status: Get "http://[::]:8200/v1/sys/seal-status": dial tcp [::]:8200: connect: connection refused

I don't think the Vault is sealed, but if I try to unseal it:

vault operator unseal
Unseal Key (will be hidden):
Error unsealing: Put "http://[::]:8200/v1/sys/unseal": dial tcp [::]:8200: connect: connection refused

Any ideas?

Solution
    1. As pertains to the tutorial, it works. Not sure what I was doing wrong, but I ran it all again and it worked. If I had to guess, I would suspect that some of the YAML involved in configuring the pods got malformed (since white space is significant).
    2. The vault status command works, but only from a terminal running inside the Vault pod. The Kubernetes-in-Docker-on-DockerDesktop cluster does not expose any ports for these pods, so even though I have vault-cli installed on my PC, I cannot use vault status from outside the pods.