When using Google Data Prep, I am able to create automated schedules to run jobs that update my BigQuery tables.
However, this seems pointless when considering that the data used in Prep is updated by manually dragging and dropping CSVs (or JSON, xlsx, whatever) into the data storage bucket.
I have attempted to search for a definitive way of updating this bucket automatically with files that are regularly updated on my PC, but there seems to be no best-practice solution that I can find.
How should one go about doing this efficiently and effectively?
So, in order to upload files from your computer to Google Cloud Storage, there are a few possibilities. If you just run an daemon process which handles any change in that shared directory, you can code an automatic upload in this different languages: C#, Go, Java, Node.JS, PHP, Python or Ruby.
You have here some code examples for uploading objects but, be aware that there is also a detailed Cloud Storage Client Libraries references and you can also find the GitHub links in "Additional Resources".