Search code examples
performanceprometheusmetrics

Prometheus does not scrape all metrics from PCP pmproxy

On my laptop with Fedora 30 I have Performance Co-Pilot (PCP) daemons installed and running, and Prometheus installed from the package golang-github-prometheus-prometheus-1.8.0-4.fc30.x86_64. In PCP collector's config I specified the following metric namespaces:

# Performance Metrics Domain Specifications
# 
# This file is automatically generated during the build
# Name  Id      IPC     IPC Params      File/Cmd
#root   1   pipe    binary      /var/lib/pcp/pmdas/root/pmdaroot
#pmcd   2   dso pmcd_init   /var/lib/pcp/pmdas/pmcd/pmda_pmcd.so
proc    3   pipe    binary      /var/lib/pcp/pmdas/proc/pmdaproc -d 3
#xfs    11  pipe    binary      /var/lib/pcp/pmdas/xfs/pmdaxfs -d 11
linux   60  pipe    binary      /var/lib/pcp/pmdas/linux/pmdalinux
#pmproxy    4   dso pmproxy_init    /var/lib/pcp/pmdas/mmv/pmda_mmv.so
mmv 70  dso mmv_init    /var/lib/pcp/pmdas/mmv/pmda_mmv.so
#jbd2   122 dso jbd2_init   /var/lib/pcp/pmdas/jbd2/pmda_jbd2.so
#kvm    95  pipe    binary      /var/lib/pcp/pmdas/kvm/pmdakvm -d 95

[access]
disallow ".*" : store;
disallow ":*" : store;
allow "local:*" : all;

When I visit the URL localhost:44323/metrics the output is very rich and covers many namespaces, ie. mem, network, kernel, filesys, hotproc etc., but when I scrape it with Prometheus where the job is defined as:

scrape_configs:
  - job_name: 'pcp'
    scrape_interval: 10s
    sample_limit: 0
    static_configs:
      - targets: ['127.0.0.1:44323']

I see the target status UP, but in the console only these two metric namespaces are available for querying: hinv and mem. I tried to copy other metric names from /metrics page, but queries result in the error: 'No datapoints found.' Initially I thought the problem could have been due to a limit on the number of samples per target or the sampling interval too small (I originally set it to 1s), but hinv and mem are not next to each other and there are other metrics (ie. filesys, kernel) in between them that are omitted. What could be the reason for that?

Solution

  • I have not found the exact cause of the problem, but it must have been version-specific, because after I downloaded and launched the latest version 2.19 the problem was gone and with exactly the same config Prometheus was reading all metrics from the target.