Search code examples
google-cloud-platformgoogle-cloud-functionshttp-headersauthorizationgoogle-oauth

Finding bearer tokens being modified with SIGNATURE_REMOVED_BY_GOOGLE in GCP functions


Ordinarily would not ask this but I have searched high and low for information on "SIGNATURE_REMOVED_BY_GOOGLE" and come up short.

I am running a webapp in firebase using GCP functions as a backend. As part of this work I use access token based authorization where I send a header that is like this:

Authorization: Bearer ${SOME ID TOKEN}

The ID token broadly breaks down into 3 parts:

`${BASE64 encoded header JSON}.${BASE64 encoded body JSON}.${signature}`

These ID tokens are generated by google from the https://securetoken.googleapis.com/v1/token endpoint

As of 2022/08/04 at around 6am GMT the GCP function started throwing errors, it had been working without issue for months.

The errors were a result of the Authorization header being modified in transit by google

from:

Authorization: Bearer ${BASE64 encoded header JSON}.${BASE64 encoded body JSON}.${signature}

to:

Authorization: bearer ${BASE64 encoded header JSON}.${BASE64 encoded body JSON}.SIGNATURE_REMOVED_BY_GOOGLE

Note the word Bearer => bearer, which while technically valid is unusual for the auth header

Primarily, the issue is the removal of the signature from the token. and it's replacement with "SIGNATURE_REMOVED_BY_GOOGLE"

I have found this SO answer: Google Cloud Run masks Bearer token in Authorization header

Which suggests that Cloud Run does this as a security precaution. However I am using GCP functions, not Cloud Run, also, this started happening today after having not happened before. That SO link is several years old so I would have expected it to be occurring before 2022/08/04 if it were caused by the same thing.

So, several questions:

  1. Is there a way to prevent the header from being modified? (I don't like it but if I used something like 'x-custom-auth' as a header would that prevent it?)
  2. If it cannot be stopped, can I assume that a request with this bearer token mapping is valid / secure?
  3. How do I Differentiate between google mapping the auth header vs some malicious actor sending a request without a signature and putting "SIGNATURE_REMOVED_BY_GOOGLE" in themselves?

Solution

  • I wrote the answer that you are referencing in your question.

    Note the word Bearer => bearer, which while technically valid is unusual for the auth header

    HTTP headers consist of case-insensitive names. Google often converts HTTP header names (keys) to lowercase.

    Is there a way to prevent the header from being modified? (I don't like it but if I used something like 'x-custom-auth' as a header would that prevent it?)

    No, you cannot prevent Google from modifying the token. The token is modified to prevent the token from being used again (to prevent the service from using the token to impersonate the user to call another service).

    Yes, you can use a custom header. However, there have been issues (bugs) reported when doing so.

    If it cannot be stopped, can I assume that a request with this bearer token mapping is valid / secure?

    If your service has IAP enabled (the service is not public), then yes, you can safely assume that the token is valid. If the client forges an identity token, IAP will reject their request.

    How do I Differentiate between google mapping the auth header vs some malicious actor sending a request without a signature and putting "SIGNATURE_REMOVED_BY_GOOGLE" in themselves?

    If IAP is enabled, a malicious actor will not get past IAP. The Identity Token signature is created with the identity's private key. The bad actor would need that private key to create requests. The signature is validated with the corresponding public key. Requests with an invalid signature are rejected with an HTTP 40x status code. SIGNATURE_REMOVED_BY_GOOGLE would be an invalid signature.

    That SO link is several years old so I would have expected it to be occurring before 2022/08/04 if it were caused by the same thing.

    New features are often released in one service and when found useful added to other services. Cloud Functions has existed for a longer time than Cloud Run. Changes to existing services sometimes takes a long time. First customers are notified that in the future XYZ will be enabled, start working, etc. I am not sure of the exact status of the enablement of OAuth token redaction for Cloud Functions.