Search code examples
python-3.xdockerraspberry-piinfluxdb

Missing data while data writing to the database


I have program which writing data to the InfluxDB every second and I found some error. Sometimes one second missing in DB. It is not regularly data missing but it is about one missing second per 1 minute.

My program looks this:

def monitoring():           
try:
    client = InfluxDBClient(host=IpAdress, port="8086", database="DB")
    print("Connected to InfluxDB")      
except:
    print('{} :Can`t connect to the database, with error {}'.format(datetime.now().strftime("%Y-%m-%d %H:%M:%S"), exc_info()[0]))

#=============================================== GPIO ==========================================
GPIO.setwarnings(False)                     # Ignore warning for now
GPIO.setmode(GPIO.BCM)                    # Use physical pin numbering
GPIO.setup(27, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                 
GPIO.setup(18, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                 
GPIO.setup(25, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                
GPIO.setup(12, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                
GPIO.setup(22, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                
GPIO.setup(19, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                 
GPIO.setup(17, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                 
GPIO.setup(23, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                
GPIO.setup(24, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                
GPIO.setup(26, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                
GPIO.setup(16, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                
GPIO.setup(20, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)                 
GPIO.setup(21, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)  
last_update = int(time.time())
while True:
    if GPIO.input(27):
        run_01 = 1
    elif GPIO.input(18):
        run_01 = 1
    else:                
        run_01 = 0
#checking all GPIO if they are UP or DOWN
    now_time = int(time.time())
    if last_update < now_time:        
        #===============================================Write_Data==========================================
        json_writer = [ 
            {
                "measurement": "measurement",
                "tags": {
                    "some_tag": "{}".format(some_tag)
                },
                    "time": datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f"), 
                    "fields": {
                        "data_to_DB": data_to_DB
                    }
            }
        ]
        try:
            client.write_points(json_writer)
        except:
            print('{} :Writing data to database error: {}'.format(datetime.now().strftime("%Y-%m-%d %H:%M:%S"), exc_info()[0]))
        last_update = now_time
client.close()

The program run on Rpi 3 or Rpi 4 inside Docker container with Python 3.7.3 and InfluxDB 1.7.7

I think it is all, but if you need more specific information please write to me and I will upgrade my question.

Do you have some hint how to resolve this problem?

EDIT:

Raw timestamps data, another column are just some strings and 0 or 1:

1580460015000029184
1580460016000039168
1580460017000029184
1580460018000050176
1580460019000039936
1580460020000034048
1580460021000029184
1580460022000039168
1580460023000036864
1580460024000029952
1580460025000036096
1580460026000036096
1580460027000029184
1580460028000050176
1580460029000044032
1580460032878979072
1580460033127377152
1580460034000029184
1580460035000040960
1580460036000046848
1580460037000040960
1580460038000033024
1580460039000033024
1580460040000033024
1580460041000026880
1580460042000040960
1580460043000026112
1580460044000036096
1580460045000039168
1580460046000039936
1580460047000036096
1580460048000043008
1580460049000026880
1580460050000049152
1580460051000032000
1580460052000045056
1580460053000036096
1580460054000039168
1580460055000029952
1580460056000039168
1580460057000033024
1580460058000046848
1580460059000049152
1580460060000035072
1580460061000036096
1580460062000035072
1580460063000039168
1580460064000029952
1580460065000039168
1580460066000039168
1580460067000043008
1580460068000036864
1580460069000049152
1580460070000036096
1580460071000029184
1580460072000040960
1580460073000043008
1580460074000033024

Solution

  • Thanks for a hint.

    I tried all your advice.

    The main problem is: sometimes the record takes about 3 seconds. Normally is about 0.03 second.

    Otherwise, I resolve this problem by moving client.write_points(json_writer) to another thread.