Extracting Small amount of Heavy records from BigQueyr

Question

I want to load 1 record that has 1.8MB of data from BigQuery query result to Java, currently I am using code like

@Autowired
BigQuery bigQuery;
         
QueryJobConfiguration queryJobConfiguration = QueryJobConfiguration.newBuilder(query)
                .setDefaultDataset(bigQueryProperties.getDatasetName())
                .setUseQueryCache(true)
                .setNamedParameters(queryParametersMap)
                .setJobTimeoutMs(jobTimeoutMs)
                .build();

bigQuery.query(queryJobConfiguration)

I know this code is better to be used for a large number of lightweight records So I am interested if there is any better approach in case we have just 1 but a very big record

Answer 1

You can use the BigQuery Storage Read API. According to the documentation, "the BigQuery Storage Read API provides a third option that represents an improvement over prior options. When you use the Storage Read API, structured data is sent over the wire in a binary serialization format. This allows for additional parallelism among multiple consumers for a set of results."

 

Here's an example in using the client library.