Apache Drill - retrieveing storage credentials from vault

Question

I am trying to setup Drill to retrieve credentials from Hashicorp Vault. I am following this docs https://github.com/apache/drill/blob/master/docs/dev/PluginCredentialsProvider.md#using-credentials-managed-by-vault and I setup everything but it's not working. I turned on DEBUG logs and this is what I'm getting:

08:27:22.496 [qtp681948432-30] DEBUG o.a.d.e.s.s.v.VaultCredentialsProvider - Attempting to fetch secrets from Vault path kv/pg_gen.
08:27:22.497 [qtp681948432-30] WARN  o.a.d.e.s.s.v.VaultCredentialsProvider - No credentials matching the configured property names were readable at kv/pg_gen
The server requested SCRAM-based authentication, but no password was provided.

It's not a problem on the vault side, as I can use their API and successfully retrieve the same secret. This is part of the storage configuration related to vault:

  "credentialsProvider": {
    "credentialsProviderType": "VaultCredentialsProvider",
    "secretPath": "kv/pg_gen",
    "propertyNames": {
      "username": "usernameSecret",
      "password": "usernamePassword"
    }
  },

This is how the secret looks like in vault:

========= Data =========
Key               Value
---               -----
passwordSecret    <pass>
usernameSecret    <user>

If I edit the storage, remove credentialsProvider and add username and password then it's working. Any hints how to make it work with vault?

UPDATE:

This is happening because secrets in my vault are created using engine V1 but Drill (actually vault-java-driver 5.1.0) expects V2 because it appends "data" to secretPath. See https://developer.hashicorp.com/vault/tutorials/secrets-management/compare-kv-versions. However, if I change that and create secret using engine V2 it still doesn't work! Reason is that response "data" key is now nested in another "data" key and VaultCredentialsProvider doesn't understand that. Response in V1:

"data": {
    "foo": "bar",
  }

Response in V2:

 "data": {
    "data": {
      "foo": "bar"
    },
...
  }

Is there a way to configure it so that VaultCredentialsProvider can correctly extract credentials? It just doesn't seem to work.

Answer 1

I solved this by using secret engine V2 and manually retrieving vault token and setting VAULT_TOKEN env variable.