How to setup staging, pre-prod for google dataflow jobs?

Question

Say we have a dataflow job:

1. Written in Apache Beam Java SDK which is a Gradle project.
2. Uses pubsub stream as input, writes results to bigtable and writes logs to BigQuery.

As with deploying a server, we can easily have a staging, pre-prod and prod environment.

I want to know what is the "idiomatic" way to do this in Apache Beam & dataflow.

Answer 1

The official google docs have an article on this which answered a lot of my questions.

Here is the article:

https://cloud.google.com/architecture/building-production-ready-data-pipelines-using-dataflow-deploying