I am implementing the standard implementation of Google Cloud Platforms- Data Loss Prevention API in Python for De-Identifying text. This is from the example in https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/dlp/deid.py for the method deidentify_with_fpe
When I run the code with the parameters I mentioned below I am getting the following error.
Can someone suggest how to fix the error?
google.api_core.exceptions.InvalidArgument: 400 Received the following error message from Cloud KMS when unwrapping KmsWrappedCryptoKey "projects/xxxx-xxx/locations/global/keyRings/dlp-key-ring-4/cryptoKeys/key9": Decryption failed: the ciphertext is invalid.
the entire trace is
Last login: Fri Feb 28 15:29:09 on ttys001
Restored session: Fri 28 Feb 2020 15:27:53 AEDT
xxxxs-MacBook-Pro:poc-bucket-flow-dlp xxxxxxxx$ python dlp3.py --verbosity=debug
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "Received the following error message from Cloud KMS when unwrapping KmsWrappedCryptoKey "projects/xxxx-xxx/locations/global/keyRings/dlp-key-ring-4/cryptoKeys/key9": Decryption failed: the ciphertext is invalid."
debug_error_string = "{"created":"@1582865236.411224000","description":"Error received from peer ipv4:1xx.2xx.1xx.1xx:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Received the following error message from Cloud KMS when unwrapping KmsWrappedCryptoKey "projects/xxxx-xxx/locations/global/keyRings/dlp-key-ring-4/cryptoKeys/key9": Decryption failed: the ciphertext is invalid.","grpc_status":3}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "dlp3.py", line 249, in <module>
main()
File "dlp3.py", line 246, in main
test_deidentify_with_fpe()
File "dlp3.py", line 240, in test_deidentify_with_fpe
key_name=KEY_NAME,
File "dlp3.py", line 101, in deidentify_with_fpe
item=item,
File "/usr/local/lib/python3.7/site-packages/google/cloud/dlp_v2/gapic/dlp_service_client.py", line 655, in deidentify_content
request, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Received the following error message from Cloud KMS when unwrapping KmsWrappedCryptoKey "projects/xxxx-xxx/locations/global/keyRings/dlp-key-ring-4/cryptoKeys/key9": Decryption failed: the ciphertext is invalid.
xxxxs-MacBook-Pro:poc-bucket-flow-dlp xxxxxxxx$ python dlp3.py --verbosity=debug
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "Received the following error message from Cloud KMS when unwrapping KmsWrappedCryptoKey "projects/xxxx-xxx/locations/global/keyRings/dlp-key-ring-4/cryptoKeys/key9": Decryption failed: the ciphertext is invalid."
debug_error_string = "{"created":"@1583026339.185480000","description":"Error received from peer ipv4:216.58.203.106:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Received the following error message from Cloud KMS when unwrapping KmsWrappedCryptoKey "projects/xxxx-xxx/locations/global/keyRings/dlp-key-ring-4/cryptoKeys/key9": Decryption failed: the ciphertext is invalid.","grpc_status":3}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "dlp3.py", line 249, in <module>
main()
File "dlp3.py", line 246, in main
test_deidentify_with_fpe()
File "dlp3.py", line 240, in test_deidentify_with_fpe
key_name=KEY_NAME,
File "dlp3.py", line 101, in deidentify_with_fpe
item=item,
File "/usr/local/lib/python3.7/site-packages/google/cloud/dlp_v2/gapic/dlp_service_client.py", line 655, in deidentify_content
request, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Received the following error message from Cloud KMS when unwrapping KmsWrappedCryptoKey "projects/xxxx-xxx/locations/global/keyRings/dlp-key-ring-4/cryptoKeys/key9": Decryption failed: the ciphertext is invalid.
I am passing the following parameters
For key_name I passed the parameter in the format (THE project name here is masked)
- key_name=(
"projects/xxxxxx/locations/global/keyRings/"
"dlp-key-ring-4/cryptoKeys/key9"
)
The wrapped text passed is in the following parameter, the key characters are masked
- wrapped_key=(
"MIIxxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
"D+Ixxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
"MIIxxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
"D+Ixxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
"MIIxxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
"MIIxxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
"D+Ixxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
"MIIxxxxxyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
"yt1zhbQNhsLICCFMNDFJDBFHBDFVBSHDFNSSKSKSKDKLSDKLMBAAE="
),
here is the code snippet
# [START dlp_deidentify_fpe]
def deidentify_with_fpe(
project,
string,
info_types,
alphabet=None,
surrogate_type=None,
key_name=None,
wrapped_key=None,
):
"""Uses the Data Loss Prevention API to deidentify sensitive data in a
string using Format Preserving Encryption (FPE).
Args:
project: The Google Cloud project id to use as a parent resource.
item: The string to deidentify (will be treated as text).
alphabet: The set of characters to replace sensitive ones with. For
more information, see https://cloud.google.com/dlp/docs/reference/
rest/v2beta2/organizations.deidentifyTemplates#ffxcommonnativealphabet
surrogate_type: The name of the surrogate custom info type to use. Only
necessary if you want to reverse the deidentification process. Can
be essentially any arbitrary string, as long as it doesn't appear
in your dataset otherwise.
key_name: The name of the Cloud KMS key used to encrypt ('wrap') the
AES-256 key. Example:
key_name = 'projects/YOUR_GCLOUD_PROJECT/locations/YOUR_LOCATION/
keyRings/YOUR_KEYRING_NAME/cryptoKeys/YOUR_KEY_NAME'
wrapped_key: The encrypted ('wrapped') AES-256 key to use. This key
should be encrypted using the Cloud KMS key specified by key_name.
Returns:
None; the response from the API is printed to the terminal.
"""
# Import the client library
import google.cloud.dlp
# Instantiate a client
dlp = google.cloud.dlp_v2.DlpServiceClient()
# Convert the project id into a full resource id.
parent = dlp.project_path(project)
# The wrapped key is base64-encoded, but the library expects a binary
# string, so decode it here.
import base64
wrapped_key = base64.b64decode(wrapped_key)
# Construct FPE configuration dictionary
crypto_replace_ffx_fpe_config = {
"crypto_key": {
"kms_wrapped": {
"wrapped_key": wrapped_key,
"crypto_key_name": key_name,
}
},
"common_alphabet": alphabet,
}
# Add surrogate type
if surrogate_type:
crypto_replace_ffx_fpe_config["surrogate_info_type"] = {
"name": surrogate_type
}
# Construct inspect configuration dictionary
inspect_config = {
"info_types": [{"name": info_type} for info_type in info_types]
}
# Construct deidentify configuration dictionary
deidentify_config = {
"info_type_transformations": {
"transformations": [
{
"primitive_transformation": {
"crypto_replace_ffx_fpe_config": crypto_replace_ffx_fpe_config
}
}
]
}
}
# Convert string to item
item = {"value": string}
# Call the API
response = dlp.deidentify_content(
parent,
inspect_config=inspect_config,
deidentify_config=deidentify_config,
item=item,
#location_id="us-east1",
)
# Print results
print(response.item.value)
# [END dlp_deidentify_fpe]
Most likely, the key is not wrapped correctly. Can you please try the following commands to see that the decryption of the wrapped key generates the right output?
cat wrapped_key.txt | base64 -d > ciphertext.txt
gcloud kms decrypt --location global --keyring <key-ring-name> --key <key-name> --plaintext-file unwrapped_secret.txt --ciphertext-file ciphertext.txt
stat --printf="%s\n" unwrapped_secret.txt
Here are the steps to generate the KMS wrapped key for use with Google Cloud DLP API.
Basic Terminology:
DEK: Key to be wrapped.
KEK: Key with which DEK would be wrapped. This key does not leave Google Cloud KMS.
Go to your Google cloud console project > cryptographic keys and create a keyring and a KEK (if not already done so).
Commands
In order to execute the following commands, Google Cloud Shell might be the best option as it takes care of gcloud credential setup for you.
Step 1. Generate a 32 Byte random value. This would be your DEK.
openssl rand 32 > secret.txt
Step 2. Encrypt using Cloud KMS
gcloud kms encrypt --location global --keyring <key-ring-name> --key \
<key-name> --plaintext-file secret.txt --ciphertext-file \
mysecret.txt.encrypted
Step 3: Convert to base64
base64 mysecret.txt.encrypted
Step 4: Use this generated value in your request to Google Cloud DLP API.
Let me know if this doesn't help. If you could provide details on how you are wrapping the key, we could triage this further.