Here is the problem I am trying to solve. Client has 1000 websites sending data layer data into GTM. Ultimately the data needs to be sent to Bigquery. So, its large amount of data and it is currently being sent to a cloud functions which then inserts it into Bigquery and due to its quotas and limits there are timeout errors.
But this approach increases the number of cloud function instances heavily, which is not cost effective. Another approach could be to send the GTM events to Google cloud pubsub and then to cloud function(where we can now control the max. no of instances). But the issue with this approach is that GTM cannot send data to an endpoint that requires authentication. Which is the case with pubsub endpoint(it needs either an API key/Oauth 2.0).
Any advice/suggestion/approach to handle this problem would be highly appreciated.
I recommend you to try Cloud Run. it's very similar to Cloud Function (wrap your function in a webserver, that's all! I wrote an article on that) and you can handle up to 250 query concurrently on the same instance (and thus reduce the cost and increase the efficiency).
I didn't catch the BigQuery issue (limit and quota), but the right way is to write in stream to BigQuery (you are limited to 1 millions write per second in stream).