Search code examples
google-bigquerygoogle-cloud-datastore

How to export from BigQuery to Datastore?

I have tables in BigQuery which I want to export and import in Datastore. How to achieve that?

Solution

  • Table from BigQuery can be exported and imported to your datastore.

    Download the jar file from https://github.com/yu-iskw/bigquery-to-datastore/releases

    Then run the command 

    java -cp bigquery-to-datastore-bundled-0.5.1.jar com.github.yuiskw.beam.BigQuery2Datastore --project=yourprojectId --runner=DataflowRunner  --inputBigQueryDataset=datastore  --inputBigQueryTable=metainfo_internal_2 --outputDatastoreNamespace=default --outputDatastoreKind=meta_internal  --keyColumn=key --indexedColumns=column1,column2 --tempLocation=gs://gsheetbackup_live/temp  --gcpTempLocation=gs://gsheetlogfile_live/temp

    --tempLocation and --gcpTempLocation are valid cloud storage bucket urls.

    --keyColumn=key - the key here is the unique field on your big query table