Search code examples
google-bigquery

Max number of partitions that can be created on bigquery table


Im getting below error while moving data from unpartitioned to partitioned table.

"Resources exceeded during query execution: Table mydataset.mytable$20101213 will have 2501 partitions when the job finishes, exceeding limit 2500..'}]"

--> Whether bigquery limits on the max number of partitions that can be created on a table ? or whether this is just a day quota ?

Moreover, this limit is not mentioned in their BigQuery "Quotas and Limits" pages


Solution

  • It's mentioned in the docs. Currently, you cannot have more than 10,000 partitions per table. You'll need to split your data/partitions across multiple tables (each of around 7 years of data).

    Each partitioned table can have up to 10,000 partitions.

    https://cloud.google.com/bigquery/quotas#partitioned_tables