Working with Jobs in BigQuery Connector in WSO2 EI 6.5.0 - maxResults limit in runQuery Operation

Question

I am using BigQuery connector in WSO2 EI 6.5.0

There is an operation called runQuery to fetch data from bigquery table.

Below is the usage of maxResult as per this Google doc,

maxResult - The maximum number of rows of data (results) to return per page. Responses are also limited to 10 MB. By default, there is no maximum row count and only the byte limit applies.

• Is there any default limit will be applied if i'm not passed this param? if yes how many rows will return after execute runQuery?
• if not, what is the max limit I can pass to this maxResult param?

For POC purpose, given maxresults as 10000. When it comes to PRODUCTION we may get more than 100000 of records(As per estimation 300000 - 400000 records)

Not sure in runtime what happened, this row count may increase.

So what is the solution for this situation? Kindly clarify on the same.

Answer 1

As you can see in the code WSO2 will not send this parameter if you don't set it. So WSO2 is not setting any defaults here, hence whatever limit is enforced by Bigquery will apply to the request. As per the Bigquery documentation, it states By default, there is no maximum row count, and only the byte limit applies. So the answer is there is no default limit.

As per the maximum limit, this varies on the dataset I believe, as per the documentation it's best to have a smaller number that will paginate the dataset and then you can page through them. This will improve reliability and probably resource exhaustion as well.

From Google doc.

Setting this flag to a small value such as 1000 and then paging through results might improve reliability when the query result set is large.