Moving a collection (and everything contained within it) to another firebase project

Question

It seems that this question, has in many different manners been asked (most promising were this post and this one. However, they are either above my current familiarity with google services or formulated in a manner where I'm not confident this is the same problem. As I can't imagine being the only one having to do this, here is the question:

I have two Firestore projects:

1. One for development.
2. One for my users.

There is a certain collection that contains documents, that contain collections... on which I work in the dev project and now, it feels ready to be moved to the other project.

How can that be done?

(I can't imagine that waiting for an improved version of a handcrafted dataset to be errorfree before giving it to the users is a rare scenario.)

Here is what I've done so far:

• Used the import/export feature:
  1. Created a bucket in dev project, created a bucket and exported the collection there (all using Googles UI, not command line)
  2. Switched projects, went to Google Cloud storage tab, created a bucket.
  3. Clicked on the bucket, and pressed "Transfer data" and then "Transfer data in".
  4. Find the dev project in the dialog, find the exported collection, select it and move it to the bucket. Leave all options as default and started the job.
  5. Go back to firebase interface, let it send me to Google Cloud Firebase (why is that so cumbersome to access?) and then import from the bucket the desired collection (for some reason I had to tweak the name for it to work).
  6. Finally, I can see the collection in my projects firestore database.
  7. I was Happy. Until I realized that all the subcollections were missing!
• Tried some things with the CLI but was unsuccessful .
• Enabled BigQuery in both projects but found it confusing.
• Tried to install "Firefoo" which claims to do exactly that but the installer is broken.
• I have given up due to the uncertainty of the subcollections and documents containing them being moved as well on exports.

So what is the proper way of doing that? (Or are you not meant to move work on some data in your dev project to then move it to prod when its ready?)

And why does it seem so complicated?

Answer 1

Update: Looking back, using a batch write or a transaction is probably a more reliable of handling this issue, however my current approach worked for me:

I have written the following script that "solved" my issue. These were my exact steps:

Follow these if you want the collection in dev (and the contained data it contains) to replace the corresponding collection in your other project.

1. In my project folder, I have created new folder movedata where I created two additional folders: source and destination.
2. In the dev firebase project, in Project Settings > Service Accounts. I generate a new private key (this downloads a file) that I put inside of the source folder.
3. In the other firebase project, I repeat step 2, only I put the file inside the destination folder.
4. I create in the folder movedata from step 1 a file called move_firestore_collection.js where I put the following script (set the COLLECTION_NAME variable with your collection if you want to use it).

const admin = require('firebase-admin');
const serviceAccountSource = require('./source/serviceAccountKey.json');
const serviceAccountTarget = require('./target/serviceAccountKey.json');

//Replace 'your-collection-id' with your actual collection.
const COLLECTION_NAME = 'your-collection-id';

// Initialize Firebase Admin SDK for source project
const sourceApp = admin.initializeApp({
  credential: admin.credential.cert(serviceAccountSource),
}, 'sourceApp');

// Initialize Firebase Admin SDK for target project
const targetApp = admin.initializeApp({
  credential: admin.credential.cert(serviceAccountTarget),
}, 'targetApp');

const sourceFirestore = sourceApp.firestore();
const targetFirestore = targetApp.firestore();

async function copyCollection(sourceFirestore, targetFirestore, collectionPath) {
  const collectionRef = sourceFirestore.collection(collectionPath);
  const snapshot = await collectionRef.get();

  for (const doc of snapshot.docs) {
    const docData = doc.data();
    await targetFirestore.collection(collectionPath).doc(doc.id).set(docData);

    // Recursively copy subcollections
    const subcollections = await doc.ref.listCollections();
    for (const subcollection of subcollections) {
      await copyCollection(sourceFirestore, targetFirestore, `${collectionPath}/${doc.id}/${subcollection.id}`);
    }
  }
}

async function main() {
  try {
    console.log(`Starting to copy collection ${COLLECTION_NAME}`);
    await copyCollection(sourceFirestore, targetFirestore, COLLECTION_NAME);
    console.log(`Successfully copied collection ${COLLECTION_NAME}`);
  } catch (error) {
    console.error('Error copying collection:', error);
  } finally {
    sourceApp.delete();
    targetApp.delete();
  }
}

main();

5. Run node movedata/move_firestore_collection.js and you should be set. The collection and nested data from dev will replace the one from your other project. If this doesn't work, try running npm install firebase-admin first, and then rerun the script.

There might be better/smarter ways of doing that, but this worked for me.