pygal plotting dates on the x-axis

Question

(Note: I am mostly new at Python so my code is more functional than elegant, or so I hope it becomes functional.)

Using Python 3.6 and pygal 2 I am attempting to read in json formated data from a storage array's REST API. I want to pull out the latency values (read, write, total) and create a scatter plot showing the values.

I started with the pygal docs about plotting dates, here.

The timestamps in json are all in epoch milliseconds. So as I read them in, I have to divide to make them work with datetime library.

Sample json data section:

    [
  1532908042508,
  {
    "latency.total": 1.09258,
    "partialBlocksRatio.total": 18.58528,
    "iops.read": 5984.2,
    "latency.read": 1.1011,
    "iops.write": 2181.4,
    "throughput.read": 1461.03762,
    "throughput.write": 105.14331,
    "throughput.total": 1566.18092,
    "iops.total": 8165.6,
    "latency.write": 1.06919
  }
],

My issue is once I get the data is how to feed it into pygal. I start with defining the chart (taken from the pygal doc example)

I pull the data, convert the epoch milliseconds to seconds, create the time string to be in the format needed (Y, m, d, H, M, S) and put that list in a list. Next load the latency data into a list. When all the data is loaded, I add to pygal and attempt to render to a file.

if key.lower() == 'history':
    for key2, historyData in value:
        epochSec = key2 / 1000.0            # The key for each history entry is epoch time in milliseconds. Time functions in Python
                                                # appear to want seconds. So the division is needed.
       timeStamp = [datetime.fromtimestamp(epochSec).strftime('%Y, %m, 
%d, %H, %M, %S')]       # Create the list format used for pygal
       timeStamps.append(timeStamp)
       #print(timeStamp)
       for key3 in historyData:
           if key3.lower() == 'latency.read':
              perfHistory.append(historyData[key3])
              for stamp in timeStamps:
                  #print(stamp[0])
                  xy_dateLine.add("", datetime(int(stamp[0])),perfHistory)
xy_dateLine.render_to_file('scatter4-1.svc')

the error is Traceback (most recent call last): File "C:/Users/../python/XIO_DataPlotting5.py", line 57, in xy_dateLine.add("", datetime(int(stamp[0])),perfHistory) ValueError: invalid literal for int() with base 10: '2018, 07, 22, 18, 02, 07'

My feeling here is I am overlooking something simple, yet I am at a lost at what. Am I over complicating this? Is there a tutorial for pygal that my google-fu uncovering?

Answer 1

I've just spent about an hour trying to solve a closely related issue – so despite the fact that this is old, I thought that I'd share my solution...

Despite the fact that PyGal documentation doesn't make it terribly clear – PyGal does in fact directly support time/dates in UNIX epoch time...

So for example to plot three points (picking numbers I was playing with for something else) in 2014 – then the code is just:

    dateline.add('Epoch Time', [
          (1389830399, 400),
          (1402826470, 450),
          (1420029285, 500)])

Or for the full example – it'd be:

import pygal
from datetime import date

dateline = pygal.DateLine(x_label_rotation=45, range=(0, 750))
dateline.x_labels = [
        date(2014, 1, 1),
        date(2014, 3, 1),
        date(2014, 5, 1),
        date(2014, 7, 1),
        date(2014, 9, 1),
        date(2014, 11, 1),
        date(2014, 12, 31)
]

dateline.add('Time', [
        (1389830399, 400),
        (1402826470, 450),
        (1420029285, 500)])

dateline.render_to_png("test.png")

Which gives you a graph that looks like this:

Anyway, I hope that's of use to save someone else the same headache that I've just had! :-)