Dockerized stackdriver agent - payloadErrors in logs

Question

We have dockerized the stackdriver agent and using /proc volume from base host to monitor processes. In the collectd.log, we are frequently getting below error:

Any clues on this issue?

    [2017-12-13 07:07:26] write_gcm: Server response (CollectdTimeseriesRequest) contains errors:
{
  "payloadErrors": [
    {
      "index": 218,
      "error": {
        "code": 3,
        "message": "Expected 4 labels. Found 0. Mismatched labels for payload [values {\n  data_source_name: \"value\"\n  data_source_type: GAUGE\n  value {\n    double_value: 483561472\n  }\n}\nstart_time {\n  seconds: 1513148845\n  nanos: 819266918\n}\nend_time {\n  seconds: 1513148845\n  nanos: 819266918\n}\nplugin: \"processes\"\nplugin_instance: \"all\"\ntype: \"ps_rss\"\n] on resource [type: \"gce_instance\"\nlabels {\n  key: \"instance_id\"\n  value: \"504805423291753409\"\n}\nlabels {\n  key: \"zone\"\n  value: \"europe-west1-b\"\n}\n] for project XXXXXXXXXXX"
      }
    }
  ]
}

Answer 1

I had the same issue and contacted Google Cloud Support. They told me that the Stackdriver service has been updated recently to accept more detailed information on ps_rss metrics, but it has caused metrics from older agents to not be sent at all.

You should be able to fix this issue by upgrading your Stackdriver agent to the latest version. I recently updated, and haven't seen the issue since.