I have a partitioned table (bigquery) and records are streamed for each date multiple times during a few days period, eg: records for 02.06.2022 are streamed on 03.06, 04.06, 05.06 and etc.
Is there a way that these updates can trigger a scheduled query and insert some of the new records (based on certain criteria) to another table? Maybe using the last modified field from the table's details somehow?
The reason: we can't wait until all updates are done before using the data that we already have but at the same time we need to have all the data that is available after the first run.
Thank you in advance!
So what i did was use TIMESTAMP_MILLIS(last_modified_time) in the where statement of the scheduled query. I made the where statement to check the last_modified_time of a list of tables that i needed. If it was after the previous run, then the query would insert the new records.