Search code examples
pythonpandasdataframedatasetdata-conversion

Create data frame with pd.read_csv but data in column is connected


The following data is a small piece of a large data set.

  -.976201  -.737468  -.338866  -.174108  -.388671  -.793479 -1.063547 -1.005576
  -.666256  -.254177   .018064   .069349  -.015640  -.090710  -.111850  -.194042
  -.486229  -.993744 -1.554215 -2.003795 -2.348716 -2.770146 -3.502312 -4.712848
 -6.401421 -8.300894 -9.896770-10.674380-10.444660 -9.438081 -8.065303 -6.594510

What I essentially want to do is to convert the data into a data frame and append a time column, however, I run into trouble on the last line in the set as the points a connected by the hyphen. This is the case in several lines in the data set but I can't figure out how to solve this problem. Eventually, I want to plot the data and therefore need to get rid of the dtype: object for the Motion column. The dataframe it gives me is shown in the appended picture and this is my code: Dataframe print

import numpy as np
import pandas as pd
time_range = np.arange(0, 500, 0.005)

motion_data = pd.read_csv('data.txt', header = None, sep = "\s+", names = range(0, 8, 1))
motion_frame = pd.DataFrame(motion_data)
motion_frame = motion_frame.stack(dropna=False).reset_index(drop=True).to_frame('Motion')
time = pd.DataFrame(time_range, index = None)
motion_frame['Time'] = time

motion_frame['Motion'].str.split('-', expand=True)
# motion_frame['Motion'].astype('float')

print(motion_frame)
motion_frame.dtypes


Solution

  • Looking at your data, every column is 10 characters wide. If it's true, you can use pandas.read_fwf() method and specify 'widths='.

    For example:

    import numpy as np
    import pandas as pd
    
    time_range = np.arange(0, 500, 0.005)
    
    motion_data = pd.read_fwf('data.txt', widths=[10] * 8, names = range(0, 8, 1))
    motion_frame = pd.DataFrame(motion_data)
    motion_frame = motion_frame.stack(dropna=False).reset_index(drop=True).to_frame('Motion')
    time = pd.DataFrame(time_range, index = None)
    motion_frame['Time'] = time
    motion_frame['Motion'] = motion_frame['Motion'].astype('float')
    
    print(motion_frame)
    print(motion_frame.dtypes)
    

    Prints:

           Motion   Time
    0   -0.976201  0.000
    1   -0.737468  0.005
    ...
    30  -8.065303  0.150
    31  -6.594510  0.155
    Motion    float64
    Time      float64
    dtype: object