I'm writing a Python script, and want to copy specific lines of a Flexible I/O (FIO) data text file to my output text file.
I've tried to copy data from my data text file using this tutorial, but sadly, I'm not getting my ideal result. After I execute my code and I concatenate into output.txt, output.txt returns nothing and not the text lines of cpu or lat (usec) from data.txt.
Here is my code to obtain data.txt:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import subprocess
import os
cmd = subprocess.Popen(["sudo", "fio", "--filename=/dev/sdg", "--name=test1", "--ioengine=libaio",
"--bs=4k", "--rw=read", "--iodepth=32", "--runtime=10"],
stdout = subprocess.PIPE, stderr = subprocess.PIPE, preexec_fn = os.setpgrp)
f = open("logfile.txt", "w+")
g = open("logs.txt", "w+")
f.write(cmd.stdout.read().decode())
with g as f:
for line in f:
# finds the leading text in the text file that tarts with "", and also removes all
# leading and trailing spaces from a strin.
if line.strip().startswith("lat (usec)") or line.strip().startswith("cpu"):
# adds items to end of list
g.write(line + "\n")
Here is my data text file (data.txt):
test1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.16
Starting 1 process
test1: (groupid=0, jobs=1): err= 0: pid=9401: Tue Jul 2 14:28:30 2019
read : io=4066.2MB, bw=416418KB/s, iops=104104, runt= 10001msec
slat (usec): min=1, max=2650, avg= 7.75, stdev=41.93
clat (usec): min=19, max=2790, avg=298.85, stdev=180.63
lat (usec): min=101, max=2792, avg=306.59, stdev=181.00
clat percentiles (usec):
| 1.00th=[ 105], 5.00th=[ 112], 10.00th=[ 118], 20.00th=[ 120],
| 30.00th=[ 129], 40.00th=[ 137], 50.00th=[ 153], 60.00th=[ 454],
| 70.00th=[ 474], 80.00th=[ 490], 90.00th=[ 502], 95.00th=[ 516],
| 99.00th=[ 548], 99.50th=[ 580], 99.90th=[ 756], 99.95th=[ 932],
| 99.99th=[ 1592]
lat (usec) : 20=0.01%, 100=0.01%, 250=51.56%, 500=36.57%, 750=11.76%
lat (usec) : 1000=0.06%
lat (msec) : 2=0.04%, 4=0.01%
cpu : usr=18.48%, sys=38.28%, ctx=16300, majf=0, minf=40
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=1041149/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: io=4066.2MB, aggrb=416417KB/s, minb=416417KB/s, maxb=416417KB/s, mint=10001msec, maxt=10001msec
Disk stats (read/write):
sdg: ios=16094/0, merge=0/0, ticks=14376/0, in_queue=14368, util=99.01%
Now I expect lines 8, 15, and 18 from the data text file into the output text file, but it's returning nothing. This is what the output text file would look like (output.txt):
lat (usec): min=101, max=2792, avg=306.59, stdev=181.00
lat (usec) : 250=5.09%, 500=67.38%, 750=23.29%, 1000=3.87%
cpu : usr=18.48%, sys=38.28%, ctx=16300, majf=0, minf=40
Is there a similar way of extracting data from my data text file into my result output file using the tutorial above?
Thank you!
I figured out my problem, thank you @AlessioM and @pkfm for your help!
Here is my answer:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import subprocess
import os
def fioCall():
cmd = subprocess.Popen(["sudo", "fio", "--filename=/dev/sdg", "--name=test1", "--ioengine=libaio",
"--bs=4k", "--rw=read", "--iodepth=32", "--runtime=10"],
stdout = subprocess.PIPE, stderr = subprocess.PIPE, preexec_fn = os.setpgrp)
f = open("data.txt", "w+")
f.write(cmd.stdout.read().decode())
f.close()
def copyFiles():
with open("data.txt") as f:
with open("output.txt", "w") as f1:
for line in f:
if line.strip().startswith("lat (usec)") or line.strip().startswith("cpu"):
f1.write(line)
def main():
fioCall()
copyFiles()
if __name__ == "__main__":
main()
and here is the output I obtained (output.txt):
lat (usec): min=161, max=10624, avg=458.78, stdev=148.10
lat (usec) : 250=5.09%, 500=67.38%, 750=23.29%, 1000=3.87%
cpu : usr=17.24%, sys=63.28%, ctx=81571, majf=0, minf=40