I am implementing the insertion to google bigquery using golang. The file has duplicates, and needs to be update everyday. However, the duplicates should not be inserted into the bigquery. I use some apis by google cloud platform and golang. However, I have not found any apis can either check if unique before inserting or setting to unique for specific fields in schema. I am using a wrapper api and calling the function directly. However, the function does not check if the field is unique. Please suggest any bigquery apis for golang has unique key checking or unique key setting to the schema. Much appreciate! I will update replies immediately.
In BigQuery, there is no any API or Function to specifically do uniqueness check of given field
You can meet your requirements by constructing your insert as join of source with destination and insert only unique ones
This might require you first to insert your data into “temp” table and then do above join in GBQ