Search code examples
pythoncsvpython-itertools

Adding fields to columns from text file using python

I have a text file which I am trying to extract columns and data from. Following is a sample of the data:

-Global stats enabled 
 Cpu Utilization : 0.1  %  45.4 Gb/core 
 Platform_factor : 1.0  
 Total-Tx        :     270.75 Mbps  
 Total-Rx        :       0.00  bps  
 Total-PPS       :      37.98 Kpps  
 Total-CPS       :       0.00  cps  

 Expected-PPS    :     102.71 Kpps  
 Expected-CPS    :       2.78 Kcps  
 Expected-BPS    :     764.51 Mbps  

 Active-flows    :      366  Clients :      252   Socket-util : 0.0023 %    
 Open-flows      :     2792  Servers :    65534   Socket :      366 Socket/Clients :  1.5 
 drop-rate       :     270.75 Mbps   
 current time    : 7.6 sec  
 test duration   : 3592.4 sec  

-Latency stats enabled 
 Cpu Utilization : 0.0 %  
 if|   tx_ok , rx_ok  , rx check ,error,       latency (usec) ,    Jitter          max window 
   |         ,        ,          ,     ,   average   ,   max  ,    (usec)                     
 ---------------------------------------------------------------------------------------------------------------- 
 0 |     1116,       0,         0,    0,          0  ,       0,       0      |  0  0  0  0  0  0  0  0  0  0  0  0  0 
 1 |     1116,       0,         0,    0,          0  ,       0,       0      |  0  0  0  0  0  0  0  0  0  0  0  0  0 
 2 |     1116,       0,         0,    0,          0  ,       0,       0      |  0  0  0  0  0  0  0  0  0  0  0  0  0 
 3 |     1116,       0,         0,    0,          0  ,       0,       0      |  0  0  0  0  0  0  0  0  0  0  0  0  0 
[2J[2H
-Per port stats table 
      ports |               0 |               1 |               2 |               3 
 -----------------------------------------------------------------------------------------
   opackets |           30391 |           48748 |           30360 |           48696 
     obytes |         2468147 |        68386300 |         2465677 |        68310324 
   ipackets |               0 |               0 |               0 |               0 
     ibytes |               0 |               0 |               0 |               0 
    ierrors |               0 |               0 |               0 |               0 
    oerrors |               0 |               0 |               0 |               0 
      Tx Bw |       4.77 Mbps |     130.69 Mbps |       4.76 Mbps |     130.53 Mbps

I need to take create columns from entries such as Total-Tx, drop-rate, etc... Then add the value for each iteration of these, into a new row.

Currently I can extract the columns, however need help to add the rows with relevant data to the csv file:

import csv
import itertools

with open('output.txt', 'r') as in_file:
    stripped = (line.strip() for line in in_file)
    lines = (line for line in stripped if line)
    grouped = itertools.izip(*[lines] * 4)
    with open('output_stats.csv', 'w') as out_file:
        writer = csv.writer(out_file)
        writer.writerow(('current time', 'drop-rate', 'Total-Tx', 'Total-Rx'))
        writer.writerows(grouped)
Solution

  • Do you want to parse and store the entries in -Global stats enabled section?

    Using regex maybe overkill, I would suggest you to use some basic string split.

    For lines with only one colon : in it (you may use str.count), using split(:) will give a list of two items, with left side as key and right side as value.

    For lines with more than one colon, you can first split by colons and spaces, with special attention to the % in Socket-util. It will give a list of [key1, value1, key2, value2, ...] (may need to flatten the list, see Making a flat list out of list of lists in Python).

    Then you can write the keys as column names, and their corresponding values.

    In case you also want to parse and store the ASCII table in -Latency stats enabled section, you can try to use the pandas, see How to create a Pandas DataFrame from String.