I'm sending GCP logs to loki and I would like to parse out the jsonPayload
as labels. When I run a query, I know I can do query
| json but is there a way to do this via the yaml config?
The jsonPayload can have different fields for different logs so I don't want to hardcode each field in the json pipeline stage
In general, having dynamic labels is not a recommended practice. More specifically it depends on the cardinality of the labels that you are sending to Loki. This article covers why picking your label set is important.
In a gist, every unique combination of key/value pairs in a label generates a new stream. These streams are persisted as chunks in your storage and Loki uses the labels in your query time as an "index" to find the proper content (chunks) to fetch.
This other post goes a bit deeper into the topic. And finally, you can take a look at the Loki label best practices.
Practically, you can add fields to the labels
action stage in the pipeline config, but it will require manually maintaining that list. AFAIK there is no way of setting all fields from the logline as labels, this may not even be easy to implement if you have a complex JSON logline (think fields with nested objects).