LogsRouter export into BigQuery results in a partial traversal down a JSON RECORD

Question

Was there was a recent change in either BigQuery or the GCP LogsRouter which stopped destructuring a JSON log after a certain depth?

Specifically, the jsonPayload.metadata.event above used to automatically get categorized as a RECORD (known from a previously setup data sink). In an attempt to fix the error, we rebuilt the tables and now both the legacy sink and our new sink has this record coming in as a STRING.

Is there any way we can have this get parsed as a RECORD again? Alternatively, if this is by design what's the recommended way to parse this now?

FWIW, it looks like the jsonPayload.metadata is getting correctly destructured which made this all the more confusing

Thanks in advance,

Answer 1

As you know the fields datatype in BigQuery are automatically created when you create a Logging Sink. You can change the data type in BigQuery as described in this BigQuery Doc

You mentioned that in an previous schema you had jsonPayload.metadata.event field as RECORD type, this means that there should be some other registries as:

jsonPayload.metadata.event.type
jsonPayload.metadata.event.name
jsonPayload.metadata.event.api_version

However in this case if you already recreated the schema it means that the datatype found for this entries is recognized as String, so before attempt any change in the BQ Schema you might want to check the logging entries that are creating the schema and in case of needed customize them. There are some documentation that might help you to achieve this.