How to store temperature and humidity data's in excel file using pandas and update that data after some time then the updated data stored in each row?

Question

I am working with ESP32 and AHT25 to measure temperature & Humidity data. I need to store that data into Excel file. I am using pandas library to store this data in csv file, My code for that is,

    pd_data = pd.DataFrame(columns=['ID','Temperature(°C)','Humidity(%)'])

    while True:
        pd_data = pd_data._append({'ID':id_json,'Temperature(°C)':temp,'Humidity(%)':humi},
                                  ignore_index=True)
        pd_data.to_csv('my_file.csv', index=False)
        time.sleep(5)

I want to store temperature and humidity data's after some time like 5 seconds and this process continuously stores data in csv file after 5 seconds in each row. But i don't know how to do this. Can anyone help me ?

I am using pandas library if any other techniques are there for store in excel sheet then suggest me that also. I will try.

Answer 1

OK, here is a solution that you can use as a starting point to achieve your goal.
Let me know if something is unclear.

1. Next time, try to post your own solution please (even if it is not working), with your thoughts, and more information.
2. Avoid using time.sleep() that is inaccurate and, above all, it blocks the execution of the program.
3. Instead of retrieving data and storing them in the .csv forever until an external interrupt occurs
   (=> while True statement) it is better to fix a max interval of time or a max number of scans as limit.
4. It is more convenient to create the .csv file only the first time and just update the same file in future use.
5. You can replace the line indicated with #** with #*** to avoid the warning: "FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead."

import time
import random
import os.path
import datetime
import pandas as pd 

csv_file = 'my_file.csv'

# Create temporary dataframe to store new readings 
temp_df = pd.DataFrame(columns=['Timestamp', 'Temperature(°C)', 'Humidity(%)'])

# Check if "my_file.csv" exists (it should be placed in the same directory)
if not os.path.isfile(csv_file):
    already = False
else:
    already = True

# Set the time limit in seconds
loop_duration = 20 
# Set the starting times 
start_time_extern = time.time()              #as float
start_time_inside = datetime.datetime.now()  #as datetime object

first_round = True
# Until the loop_duration expires
while (time.time() - start_time_extern) < loop_duration:
     current_time = datetime.datetime.now()
     elapsed_time = (current_time - start_time_inside).total_seconds()
     
     if first_round or elapsed_time >= 5:
        """ Now! and every 5 seconds: 
            _Get your data from sensors someway;
            _Do your additional operations on readings to create "temp" and "humi";
            _Add sensors' ouputs + timestamp to csv.
        N.B. Here, I've used random variables as placeholders (with 2f precision)
        N.B.2. Change to "if elapsed_time >= 5:" to wait 5 sec even the first time)
        """
        temperature = random.uniform(-10, 40)
        # Reduce the digits after the decimal point to 2
        temperature_ok = round(temperature * 100) / 100    

        humidity_percent = random.uniform(0, 100)
        humidity = humidity_percent / 100
        # Round digits as above
        humidity_ok = round(humidity * 100) / 100

        # Create current timestamp 
        timestamp = current_time.strftime('%Y-%m-%d %H:%M:%S')
        # Append to the temporary dataframe
        #**
        temp_df = temp_df.append({'Timestamp': timestamp, 
                                  'Temperature(°C)': temperature_ok, 
                                  'Humidity(%)': humidity_ok}, 
                                  ignore_index=True) 
        #***        
        #temp_df = pd.concat([temp_df, pd.DataFrame.from_records([{'Timestamp': timestamp,  
        #'Temperature(°C)': temperature_ok, 'Humidity(%)': humidity_ok}])],
        #ignore_index=True)  

        if already:
            """ If "my_file.csv" already exists...
            - Copy to csv appending data to the end of the file (mode='a' param)
              rather than overwriting any existing data.
            - Specify that the file does not add the header row with the names 
              of colums (header=False param).
            - State that the dataframe's index is not be included 
              (index=False parameter).
            """
            temp_df.to_csv(csv_file, mode='a', header=False, index=False)
            # Empty the df, to avoid to insert int the csv duplicate rows 
            temp_df = pd.DataFrame(columns=['Timestamp', 'Temperature(°C)', 'Humidity(%)'])
        else: 
            # Copy to csv including headers
            temp_df.to_csv(csv_file, index=False)
        
        # Set the starting time to now, to count again for 5 seconds 
        start_time_inside = current_time 

        first_round = False

CSV

Timestamp,Temperature(°C),Humidity(%)
2023-07-26 16:47:42,10.54,0.23
2023-07-26 16:47:47,28.19,0.71
2023-07-26 16:47:52,20.14,0.8
2023-07-26 16:47:57,-9.6,0.9
2023-07-26 16:48:10,-5.33,0.91
2023-07-26 16:48:15,21.86,0.32
2023-07-26 16:48:20,14.44,0.68
2023-07-26 16:48:25,24.9,0.59