Hashicorp Vault - Curl fails permission denied - Can't delete a Secret - Created policy with delete capability and token using policy

Question

vault --version: Vault v1.9.2

I have a policy file created, with few capabilities, especially delete:

# cat ~/.my_policy.hcl 
path "secret/*" {
  capabilities = ["create", "read", "update", "list", "delete"]
}

Created new policy using this file and I can now see the policy in the list operation:

# vault policy write my-policy ~/.my_policy.hcl 
Success! Uploaded policy: my-policy

# vault policy list
default
my-policy
root

# vault policy read my-policy
path "secret/*" {
  capabilities = ["create", "read", "update", "list", "delete"]
}

Created a new token using the above policy (so I can use it in CURL -X DELETE operation):

# vault token create -policy=my-policy; # using -no-default-policy didn't help with curl output
Key                  Value
---                  -----
token                s.27T3cNB4PrHll9byc6tppHw9
token_accessor       C6mu2crjudeHVy5jijcFkF4K
token_duration       768h
token_renewable      true
token_policies       ["default" "my-policy"]
identity_policies    []
policies             ["default" "my-policy"]

But, when I'm looking at the capabilities of the token at the folder path defined in my policy file, it shows a different policy root and shows deny

# vault token lookup s.27T3cNB4PrHll9byc6tppHw9
Key                 Value
---                 -----
accessor            C6mu2crjudeHVy5jijcFkF4K
creation_time       1678469133
creation_ttl        768h
display_name        token
entity_id           n/a
expire_time         2023-04-11T17:25:33.405537533Z
explicit_max_ttl    0s
id                  s.27T3cNB4PrHll9byc6tppHw9
issue_time          2023-03-10T17:25:33.405548806Z
meta                <nil>
num_uses            0
orphan              false
path                auth/token/create
policies            [default my-policy]
renewable           true
ttl                 767h48m50s
type                service

# These should have shown my-policy than root
# This should have shown all policies having some capability at this path
# ----

# vault token capabilities s.27T3cNB4PrHll9byc6tppHw9
root
# vault token capabilities secrets/*
root
# vault token capabilities secrets
root

# This should not give me deny when this token has the necessary policy 'my-policy' with 'delete' capability
# ----
# vault token capabilities s.27T3cNB4PrHll9byc6tppHw9 secrets/*
deny
# vault token capabilities s.27T3cNB4PrHll9byc6tppHw9 secrets
deny

Error mesg:

# curl -k -s -X GET -H 'X-Vault-Token: s.27T3cNB4PrHll9byc6tppHw9' https://vaultserver:8200/v1/secret/data/testA1/test
{"errors":["permission denied"]}

# curl -k -s -X DELETE -H 'X-Vault-Token: s.27T3cNB4PrHll9byc6tppHw9' https://vaultserver:8200/v1/secret/data/testA1/test
{"errors":["permission denied"]}

When querying vault directly using cmd line, to see secret/testA1/test secret, it spits (PS: vault API call requires/puts /data/ in the secret path):

{
  "ttl": "90d",
  "username": "test",
  "value": "KneelB4Me!YaRight"
}

Setting: the following before any of these commands has no effect.

VAULT_ADDR=https://vaultserver:8200
VAULT_NAMESPACE=admin

When creating token with -no-default-token, token capabilties at secret/* shows valid capabilities (instead of deny), still curl command fails.

Answer 1

Solution: had to pass header -H "X-Vault-Request: true", otherwise, even with 'delete' capability available in a policy, an attached to token with the right policy-plus-capability, we will still get permission denied error during CURL call.

NOTE: Passing Vault Namespace didn't help as it's Vault OSS version not paid version. Namespace doesn't work in OSS as of now.

Gotchas

1. If a secret path is: secret/testA1/test then,
2. it's API path in Vault will automatically become secret/data/testA1/test and
3. it's Metadata path in Vault will automatically become secret/metadata/testA1/test

Created a wrapper shell script, passed secret(path) from command line as secret/testA1/test

deleteSecret() {
  local metadata_secret_path=${secret_path/secret\/data/secret\/metadata}

  # NOTE: Both metadata_secret_path or secret_path path variables will work; metadata is preferred as it'll delete all versions 
  # ----: Header "X-Vault-Request: true" is imporant for delete to work, or a user will get permission denied,
  #       even with valid 'delete' capability in vault-repl policy for secret/* path

  local result=$(curl -k -s -X DELETE -H "X-Vault-Request: true" -H "X-Vault-Token: $token" $VAULT_ADDR/v1/${metadata_secret_path})
  
  if [[ $result =~ "permission denied" ]] || [[ $result =~ "errors" ]]; then
    echo "$result"
    exit 2
  fi

  echo -e "\n-- Secret deleted (including metadata): ${main_secret_path} i.e. API path: ${secret_path} or it's Metadata path: ${metadata_secret_path}\n"
}

Where the following was exported from cmd line or within wrapper script.

export VAULT_ADDR=https://vaultserver:8200

and token was generated using:

  local result=$(curl -k -s --cert $cert --key $key $VAULT_ADDR/v1/auth/cert/login -X PUT)

  if [[ $result =~ "client_token" ]]; then
    token=$(echo "$result" | jq -r '.auth.client_token')
  fi