Is there any way to track a job across services in stackdrive?

Question

We use lots of components in Google Cloud, for example a job may start on App Engine, then do some work in Apache Airflow, then do some Dataflow work which will run a BigQuery insert.

Is there any way we can track the status of a job across all components using stack driver. For example tell stackdriver somehow a custom job id and query for it.

Answer 1

You can use advanced logs filters [1] to include log entries from various products. In the logging page search for your BigQuery Job ID. Click to the Job ID and select show matching entries. This will open advanced filter text box with the proper syntax. Then you can add more queries with an OR in between.