What to pass as arguments while creating scioContext using ContextAndArgs in Scio Spotify

Question

I am new to Scio and was trying to learn more about it. I saw some examples in the Scio source code and wanted to run it. But it asks for some argument which I am unaware and are not specified in Docs.

 val (sc, args) = ContextAndArgs(cmdlineArgs)

For this part of the code, I need to pass some arguments and I am not aware what that exactly needs and in what format. Also, how can I pass Google Cloud Credential directly to the code if I don't have gsutil installed in the system?

Answer 1

GCP credentials is part of GcpOptions which extends PipelineOptions which is how Beam handles command line options. So you'll pass in --gcpCredentail in this case. https://beam.apache.org/releases/javadoc/2.7.0/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.html

There are other ways too, e.g. configure gcloud SDK to use the user's default credential, or a JSON secret file via an environment variable. https://github.com/spotify/scio/wiki/FAQ#how-do-i-fix-invalid-default-bigquery-credentials