Search code examples
dialogflow-esgoogle-cloud-logging

No json_payload for DialogFlow logs in Stackdriver in GCP

Recently, I tried to retrieve DialogFlow logs (i.e. containing everything regarding session information/interaction) from StackDriver to perform my custom analysis for dashboard purpose by enabling the option below:

enter image description here

However, after I checked the format of DialogFlow log entries, it seems there is only unstructured text_payload available (i.e.) and not having structured json_payload.

The pain is that I cannot easily parse the text_payload to get the data I want from it. Therefore, my question is:

  1. Can json_payload be available by enabling some settings that I missed?
  2. If for some reason json_payload is not available, any alternative format that can be easy to use for data retrieval? or any parser I can use onto the text_payload?

Thanks in advance.

A log entry format of a Dialogflow request:

{ "textPayload": "Dialogflow Request : {"session":"44885105","query_input":"{\n \"event\": {\n \"name\": \"WELCOME\",\n \"parameters\": {\n }\n }\n}","timezone":"Australia/Sydney"}", "insertId": "gnp3xsg27351ns", "resource": { "type": "global", "labels": { "project_id": "ai-hackathon-2020-lrwc" } }, "timestamp": "2020-08-07T04:11:29.737Z", "severity": "INFO", "labels": { "request_id": "05f6f343-a646-42e0-8181-48c2e853e21b-0820055c", "protocol": "V2", "type": "dialogflow_request" }, "logName": "projects/ai-hackathon-2020-lrwc/logs/dialogflow_agent", "trace": "44885105", "receiveTimestamp": "2020-08-07T04:11:29.800030654Z" }

A log entry format of a Dialogflow response: { "textPayload": "Dialogflow Response : id: "05f6f343-a646-42e0-8181-48c2e853e21b-0820055c"\nlang: "en"\nsession_id: "44885105"\ntimestamp: "2020-08-07T04:11:29.747Z"\nresult {\n source: "agent"\n resolved_query: "WELCOME"\n action: "input.welcome"\n score: 1.0\n parameters {\n }\n contexts {\n name: "defaultwelcomeintent-followup"\n lifespan: 2\n parameters {\n }\n }\n metadata {\n intent_id: "22498e9a-efcf-43e0-a945-36a7ef4c702d"\n intent_name: "Default Welcome Intent"\n webhook_used: "false"\n webhook_for_slot_filling_used: "false"\n is_fallback_intent: "false"\n }\n fulfillment {\n speech: "Hey Good Day! what kind of issue do you want to report？"\n messages {\n lang: "en"\n type {\n number_value: 0.0\n }\n speech {\n string_value: "Hey Good Day! what kind of issue do you want to report？"\n }\n }\n }\n}\nstatus {\n code: 200\n error_type: "success"\n}\n", "insertId": "gnp3xsg27351nt", "resource": { "type": "global", "labels": { "project_id": "ai-hackathon-2020-lrwc" } }, "timestamp": "2020-08-07T04:11:29.789Z", "severity": "INFO", "labels": { "type": "dialogflow_response", "protocol": "V2", "request_id": "05f6f343-a646-42e0-8181-48c2e853e21b-0820055c" }, "logName": "projects/ai-hackathon-2020-lrwc/logs/dialogflow_agent", "trace": "44885105", "receiveTimestamp": "2020-08-07T04:11:29.800030654Z" }

Solution

  • I had a look at this document on Processing payloads and found when the log line is a serialized JSON object and the detect_json option is enabled, the output plugin transforms the log entry into a structured (JSON) payload. By default this option is enabled for VM instances running on the App Engine flexible environment and Google Kubernetes Engine.Whereas this is not enabled by default in VM instances running on the App Engine standard environment.

    However, It is possible to customize the agent's configuration to support ingesting structured logs from additional resources. Documentation on Streaming structured (JSON) log records to Cloud Logging might be helpful.

    The payload of log records streamed by a custom-configured Logging agent can be either a single unstructured text message (textPayload) or a structured JSON message (jsonPayload).