Recommendation for architecting batch data transfer from big query to multiple cloud functions

Question

I'm currently trying to find an efficient way of reading data from a table in Big Query to then serve as inputs for my multiple cloud functions. I would then like to have the functions run some processing concurrently.

Any critiques or recommendations would be much appreciated!

Answer 1

If you perform batch processing, use Batch solution instead of real time services.

• Vertex AI Batch Prediction
• Cloud Run Jobs (with tasks in parallel)
• Batch (compute engine)
• or, my favorite because you already have the data in BigQuery, BigQuery ML with custom model

are more suitable for your needs.