How best to handle async response from google long running operation with cloud functions

Question

I'm using Google Cloud Functions (python) to initiate an asset inventory export from GCP by calling the exportAssets() method here. The method returns an Operations object defined here which can be used to poll the operation until it is complete. Of course since this is a cloud function I'm limited to 540 seconds so cannot do that forever. The google api python client offers the add_done_callback() method where one can await an async response, but as far as I can tell it requires me to keep a thread alive within the cloud function. Is there a way to tell the Asset Inventory API executing the operation to send the aync response (success or failure) to a pubsub topic where I can properly handle the response? Trying to avoid spinning up an appengine instance with basic_scaling to support 24 hour timeouts.

    from google.cloud import asset_v1
    # .....
    # Setup request to asset inventory API
    parent = "organizations/{}".format(GCP_ORGANIZATION)
    requested_type = 'RESOURCE'
    dataset = 'projects/{}/datasets/gcp_assets_{}'.format(GCP_PROJECT, requested_type)
    partition_spec = asset_v1.PartitionSpec
    partition_key = asset_v1.PartitionSpec.PartitionKey.REQUEST_TIME
    partition_spec.partition_key = asset_v1.PartitionSpec.PartitionKey.REQUEST_TIME

    output_config = asset_v1.OutputConfig()
    output_config.bigquery_destination.dataset = dataset
    output_config.bigquery_destination.table = 'assets'
    output_config.bigquery_destination.separate_tables_per_asset_type = True
    output_config.bigquery_destination.partition_spec.partition_key = partition_key

    # Make API request to asset inventory API
    print("Creating job to load 'asset types: {}' to {}".format(
        requested_type,
        dataset
    ))
    response = ASSET_CLIENT.export_assets(
        request={
            "parent": parent,
            "content_type": content_type,
            "output_config": output_config,
        }
    )
    print(response.result())  # This waits for the job to complete

Answer 1

Cloud Asset inventory export doesn't offer a PubSub notification at the end of the export. However, in my previous company, it took about 5 minutes to export 100k+ assets; it's not so bad! And if you have more assets, I'm sure you can contact Google Cloud (use your Customer Engineer) to add this notification in the roadmap.

---

Anyway, if you want to build a workaround, you can use workflows.

• Use a Cloud Function to trigger your workflow
• In your workflow,
  • Call the Cloud Asset API to export data to BigQuery
  • Get the response and perform a loop (test the export job status, if not OK, sleep X seconds and test again)
  • When the job is over, call PubSub API (or directly a Cloud Function) to submit the job status and process it.