Search code examples
pythonresourcesmetricsgraphitegraphite-carbon

Grahite, Carbon, CollectD, StatsD - Tagged Series in Python Program


I am trying to run some some python/other language modules modules/workflows/workloads and collect their CPU, Mem, I/O etc resource usage with Grahite, Carbon, CollectD, StatsD. I have read the documentation (see:here) about creating tagged series but what I can't seem to find how to tag specific moudles. For example, I have two modules

First Module

def firstModule:
    # Initialize a list
    primes = []

    for possiblePrime in range(2, 21):
       # Assume number is prime until shown it is not. 

       isPrime = True

       for num in range(2, possiblePrime):
          if possiblePrime % num == 0:
          isPrime = False

    if isPrime:
        primes.append(possiblePrime)

Second Module

def secondModule:
    # Initialize a list

    primes = []

    for possiblePrime in range(2, 21):

        # Assume number is prime until shown it is not. 

        isPrime = True

        for num in range(2, possiblePrime):
            if possiblePrime % num == 0:
            isPrime = False
            break

    if isPrime:
        primes.append(possiblePrime)

Here I want to call the two modules and then tag resource usage metrics so that I can send it to the whisper database as shown in code below: How do I achieve that?

firstModule() # assign some tag say A
secondModule() # assign some tag say B

Solution

  • Although I ended up not using a tagged series, I have solved the problem above by writing a bash script and using the render api in graphite. Here is the walk around solution:

        #!/bin/bash   
    
        #Remove previous records. Be careful not to remove useful files!
        rm cpu.json, mem.json
    
        #clear variables for storing time 
        unset start
        unset finish
    
        #log start time of process using absolute time, if using relative time check doc at https://graphite.readthedocs.io/en/latest/render_api.html for rules
        start=$(date '+%H:%M_%Y%m%d')
    
        #call first module
        firstModule()
    
        #sleep 1 minute or more because you will get an error that start time and end time are equal
        sleep 100
    
        #call second module
        secondModule()
    
        #log finish date using absolute time, if using relative time check  doc at https://graphite.readthedocs.io/en/latest/render_api.html) for rules
        finish=$(date '+%H:%M_%Y%m%d')
    
        #collect data using curl here cpu usage and memusage, adjust to fit your setup and Voila!
        curl "http://host/render?target=carbon.agents.*.cpuUsage&width=500&height=300&from={$start}&until={$finish}&format=json" > cpu.json
        sudo curl "http://host/render?target=collectdlocalhost.memory.memory-used&width=500&height=300&from={$start}&until={$finish}&format=json" > mem.json