I have a dataset of multiple .dat files but only need the last line of each. So far i can loop through my folder structure and can read the file out, but just as a whole or incorrect. And in the csv file the data is just in one cell.
I tried solutions from previous questions but i could only solve one, not both. I still dont get just the last line appended. By now he makes x (number of files) entries in my csv file, but they are neither seperated nor the right values. The values arent even at all in my .dat files.
The first column should contain the names of the entrys but as i struggled to append the last line, i had my problems here aswell, i dont know how to acces this one before the loop through my folders. I thought to convert them all to csv files but that seemed unefficient.
The code is as follows: All_Forces.csv should be the csv file I want all my data in, i created it before the loop so it doesnt get overwritten.
The .dat file contains 25 different columns seperated by blank space with 1000-10000+ lines.
from csv import writer
import pandas as pd
import os.path
df = pd.DataFrame(list())
df.to_csv('All_Forces.csv')
allV = ['V0.8','V1','V1.1']
allD = ['D1','D2','D3']
allPhi = ['Phi1','Phi2','Phi3']
allZ = ['Z1','Z2','Z3']
allT = ['T1','T2','T3']
for V in allV:
for D in allD:
for Phi in allPhi:
for Z in allZ:
for T in allT:
# example path to one file
# D:\V1\D1\Phi1\V1-T1-D1-Phi1-Z1\ForceDat
datname = "D:/" + str(V) + "/" + str(D) + "/" + str(Phi) + "/" + str(V) + "-" + str(T) + "-"+ str(D) + "-"+ str(Phi) + "-"+ str(Z) + "/ForceDat/body_0_forces.dat"
#first row entry should have this name
rowname = str(V) + "_" + str(T) + "_" + str(D) + "_" + str(Phi) + "_" + str(Z)
# not all calculations done, check if file exists
check_file = os.path.isfile(datname)
while check_file == True:
# last line
with open(datname, 'rb') as f:
lines = f.read().splitlines()
last_line = lines[-1]
List1 = last_line
#open our existing csv file in append mode
# create a file object for this file
with open('All_Forces.csv', 'a') as f_object:
writer_object = writer(f_object)
# pass the list as an argument into
# the writerow()
writer_object.writerow(List1)
# Close the file object
f_object.close()
break
import csv
import pandas as pd
import os.path
allD = ['D000','D075', 'D054', 'D068']
allPhi = ['Phi000','Phi300','Phi-300','Phi600','Phi-600']
allZ = ['Z027','Z030','Z032']
allT = ['T-300','T-450','T-600']
outfile = open('All_Forces_V0.8.csv',"a", newline='')
writer = csv.writer(outfile, delimiter =';')
for D in allD:
for Z in allZ:
for Phi in allPhi:
for T in allT:
datname = "D:/V0.8/" + str(T) + "_" + str(Phi) + "_" + str(Z) + "_" + str(D) + "/ForceDat/body_0_forces.dat"
rowname = str(T) + "_" + str(Phi) + "_" + str(Z) + "_" +str(D)
check_file = os.path.isfile(datname)
while check_file == True:
with open(datname, "r") as f:
lines = f.read().splitlines()
last_line = lines[-1]
data = last_line.split()
data = [float(s) for s in data]
if data[0] > 0.5:
data.insert(0, 'True')
else:
data.insert(0, 'false')
data.insert(0, rowname)
writer.writerow(data)
break
I solved it with this variant, if someone has ideas for improvements im open for it (still learning)