Append data in realtime to an empty pandas DataFrame

Question

I'd like to add some data, in realtime, to an empty DataFrame:

import pandas as pd
import time

df = pd.DataFrame(columns=['time', 'price'])   # this is a simple example
                                               # but in my code, I have more 
                                               # columns: 'volume', etc.
for i in range(5):                             # here it lasts one day in my real use case
    time.sleep(2)
    t = pd.datetime.now()
    df[t] = 5 + i
    # here I need to have access to the latest updates of df

print df

The output is:

Empty DataFrame  
Columns: [time, price, 2015-12-27 01:55:29.812000, 2015-12-27 01:55:31.812000, 2015-12-27 01:55:33.812000, 2015-12-27 01:55:35.812000, 2015-12-27 01:55:37.812000]  
Index: []

whereas I wanted:

time                                price
2015-12-27 01:55:29.812000          5
2015-12-27 01:55:31.812000          6
2015-12-27 01:55:33.812000          7
...

How to append data to a DataFrame like this?

Answer 1

You are indexing into the DataFrame into column t with df[t]. I think you would like to index into it by row instead.

From the looks of it though, it appears a Series may be better suited since you are updating by a time index.

import pandas as pd
import time

series = pd.Series()

for i in range(5):
    time.sleep(2)
    t = pd.datetime.now()
    series[t] = 5 + i

print series


import pandas as pd
import time

In the case that a dataframe is needed, it can be appended using df.ix[row_index]:

df = pd.DataFrame(columns = ['col1', 'col2'])

for i in range(5):
    time.sleep(2)
    t = pd.datetime.now() # Generate row index
    df.ix[t] = {'col1': 5 + i, 'col2': 20 + i}


print df