Search code examples
google-cloud-firestoregcloud

Exporting firebase data to json


I'm trying to export a particular Firestore collection using gcloud Right now, I did

gcloud storage buckets create gs://my-bucket 
gcloud firestore export gs://my-bucket --collection-ids=my-collection 
gcloud storage cp -r gs://my-bucket/2022-09-22T09:20:16_11252 my-directory

which results in some content in my-directory/all_namespaces/.../output-1. The output-1 file definitely seems to contain some relevant data but it is not very readable. Hence some questions:

  • which file format is used?
  • can I export to JSON (or CSV or XML) directly
  • can I convert the current file to JSON, CSV or XML

And related

  • why is output-0 empty?

Solution

  • Surprisingly, there don't seem to be a lot of LevelDB tools available so exporting to LevelDB format is not convenient.

    I managed to export to csv by adding two extra steps: loading and extracting to/from BigQuery. So I now do something like

    # create bucket
    gcloud storage buckets create gs://my-bucket 
    
    # firestore export
    gcloud firestore export gs://my-bucket/my-prefix --collection-ids=my-collection 
    
    # create dataset
    bq mk --dataset my_dataset
    
    # load bucket into BigQuery
    bq load --replace --source_format=DATASTORE_BACKUP my_dataset.input \
    gs://my-bucket/my-prefix/all_namespaces/.../....export_metadata
    
    # export BigQuery as csv to bucket
    bq extract --compression GZIP 'my_dataset.input' gc://my-bucket/results.csv
    
    # download csv file
    gcloud storage cp -r gc://my-bucket/results.csv <local-dir>