Search code examples
firebasegoogle-cloud-firestoregoogle-bigqueryfirebase-extensions

Firebase Biq Query extension - Exceeded rate limits: too many api requests per user per method for this user_method

I have the firebase extension for streaming data to Big Query installed https://extensions.dev/extensions/firebase/firestore-bigquery-export.

Each month I run a job to import data into my Firestore collection in batches. This month I imported 2706 rows but only 2646 made it into Big Query (60 less).

I am got the following errors from the extension: [![enter image description here][1]][1]

I contacted Firebase support and they suggested I upgrade to the latest firebase admin and function packages but these have breaking changes. Updating the latest version of firebase-admin gave me errors. I have not got any more help from them and it is still happening for multiple collections.

The options I see are:

  1. Update to the latest firebase-admin and firebase-functions packages and change my code to work with the breaking changes. I think this is unlikely to help.
  2. Update the firebase extension to the latest version from 0.1.24 to 0.1.29 which now includes a flag called "Use new query syntax for snapshots" which can be turned on. I can't find much information about this.
  3. Increase the Big Query quota somehow.
  4. Slow down the data being entered into Firestore or add it daily/weekly rather than monthly.

Here is my code in Nodejs:

  const platformFeesCollectionPath = `platformFees`;
  const limit = 500;
  let batch = db.batch();
  let totalFeeCount = 0;
  let counter = 0;

  for (const af of applicationFees) {
    const docRef = db.collection(platformFeesCollectionPath).doc();
    batch.set(docRef, { ...af, dateCreated: getTimestamp(), dateModified: getTimestamp() })

    counter++;
    if (counter === limit || counter === applicationFees.length) {
      await batch.commit();
      console.log(`Platform fees batch run for ${counter} platform fees`);
      batch = db.batch();
      totalFeeCount = totalFeeCount + counter;
      counter = 0;
    }
  }

  if (applicationFees.length > limit) {
    // Need this commit if there are multiple batches as the applicationFees.length does not work
    await batch.commit();
    totalFeeCount = totalFeeCount + counter;
  }
  
  if (counter > 0) {
    console.log(`Platform fees batch run for ${totalFeeCount} platform fees`);
  }

Update: If I look in the GCP logs using the query:

protoPayload.status.code ="7"
protoPayload.status.message: ("Quota exceeded" OR "limit")```

I can see many of these errors:
[![Errors][2]][2]


  [1]: https://i.sstatic.net/BAgTm.png
  [2]: https://i.sstatic.net/eswzI.png

Edit:
Added issue to the repo:
github.com/firebase/extensions/issues/1394

Update:
It is still not working with v0.1.29 of the bigquery extension. I am getting the same errors.
Solution

  • This was resolved in version 0.1.33 of the BQ extension. More details here: https://github.com/firebase/extensions/issues/1394