Search code examples
pythongoogle-cloud-platformgoogle-cloud-functionsgoogle-cloud-storagegoogle-cloud-composer

Trigger DAG from google cloud function

I have been trying to execute a DAG in Cloud Composer (composer-1.8.1-airflow-1.10.3) when a file arrives at a GCS bucket based on the cloud composer documentation with Cloud Function in Python

I haven't been able to make it work, every time it triggers, the function returns this message to the log

Exception: Bad response from application: 400 / {'Date': 'Mon, 30 Dec 2019 18:07:02 GMT', 'Content-Type': 'text/html', 'Content-Length': '192', 'Server': 'gunicorn/19.9.0', 'Via': '1.1 google', 'Alt-Svc': 'quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000'} / '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>400 Bad Request</title>\n<h1>Bad Request</h1>\n<p>The browser (or proxy) sent a request that this server could not understand.</p>\n'

The request URL s¡generate and the open id token seems correct The error is raised in the last part of the code

resp = requests.request(
    method, url,
    headers={'Authorization': 'Bearer {}'.format(
        google_open_id_connect_token)}, **kwargs)
if resp.status_code == 403:
    raise Exception('Service account {} does not have permission to '
                    'access the IAP-protected application.'.format(
                        signer_email))
elif resp.status_code != 200:
    raise Exception(
        'Bad response from application: {!r} / {!r} / {!r}'.format(
            resp.status_code, resp.headers, resp.text))

Thanks

Solution

  • The error code 400 Bad Request reflected in the initial question more often corresponds to the client side problem, indeed the input parameters for DAG triggering should be inspected as a first, the function that invokes IAP call in the code example:

    def make_iap_request(url, client_id, method='GET', **kwargs)

    Ensure, that you have applied dependencies required to run against IAP protected URL via requirements.txt file.

    Do not forget to verify the essential arguments that you pass to this function:

    url: The Identity-Aware Proxy-protected URL to fetch.

    client_id: The client ID used by Identity-Aware Proxy.

    So far I haven't found even better hand on experience about triggering the DAG then explained by @Jacob Ferriero in his great article.