Issue in calculating frames sent per second opencv

Question

I have the following code, I am calculating time taken to send frames per second from client to server, basically calculating the percentage of frames drop rate and time taken to respond back and communication mode is asynchronous. Now I am facing some issue in calculating the two metrics, for time taken to respond I have set delay to be more than 5 seconds because due to network and processing speed of server it takes time to send result back to client, therefore, longer delay, but for frames per second, I need to calculate how many frames are sent from the client to server per second, how would I calculate this in the data_rate method. both metrics need different time delay, I cant use same time delay for both metrics. Help is highly appreciated on how to define this in the code.

IMAGE_FOLDER = "videoframe"
FPS = 5
SERVER_A_ADDRESS = "tcp://localhost:5555"
ENDPOINT_HANDLER_ADDRESS = "tcp://*:5553"
SERVER_A_TITLE = "SERVER A"
SERVER_B_TITLE = "SERVER B"
context = zmq.Context()
socket_server_a = context.socket(zmq.PUSH)
socket_server_endpoint = context.socket(zmq.PULL)

socket_server_a.connect(SERVER_A_ADDRESS)
socket_server_endpoint.bind(ENDPOINT_HANDLER_ADDRESS)

destination = {
 "currentSocket": socket_server_a,
 "currentServersTitle": SERVER_A_TITLE,
 "currentEndpoint": SERVER_B_TITLE,}

running = True
endpoint_responses = 0
frame_requests = 0
filenames = [f"{IMAGE_FOLDER}/frame{i}.jpg" for i in range(1, 2522)]

def handle_endpoint_responses():
  global destination, running, endpoint_responses
  while running:
    endpoint_response = socket_server_endpoint.recv().decode()
    endpoint_responses += 1
def data_rate():
  global destination, running, endpoint_responses, frame_requests
  while running:
    before_received = endpoint_responses ###
    time.sleep(5)
    after_received = endpoint_responses
    before_sent = frame_requests
    time.sleep(1)
    after_sent = frame_requests ###
    print(25 * "#")
    print(f"{time.strftime('%H:%M:%S')} ( i ) : receiving model results: {round((after_received - before_received) / 5, 2)} per second.")
    print(f"{time.strftime('%H:%M:%S')} ( i ) : sending frames: {round((after_sent - before_sent) / 1, 2)} per second.")
    print(25 * "#")
def send_frame(frame, frame_requests):
  global destination, running
  try:
    frame = cv2.resize(frame, (224, 224))
    encoded, buffer = cv2.imencode('.jpg', frame)
    jpg_as_text = base64.b64encode(buffer)
    destination["currentSocket"].send(jpg_as_text)
  except Exception as Error:
    running = False
def main():
 global destination, running, frame_requests
 interval = 1 / FPS
 while running:
    for img in filenames:
        frame = cv2.imread(img)
        frame_requests += 1
        threading.Thread(target=send_frame, args=(frame, frame_requests)).start()
        time.sleep(interval)
 destination["currentSocket"].close()
if __name__ == "__main__":
  threading.Thread(target=handle_endpoint_responses).start()
  threading.Thread(target=data_rate).start()
  main()

Answer 1

Not only the server time, opening the image also takes time, using sleep interval = 1/FPS may lead to a frame drop too, i.e. producing less frames than possible (the same if playing offline). For playing, if done with sleep, the interval could be shorter and current time could be checked in the loop, and if the time is appropriate - sending the next frame, if not - just wait. The delay could be adaptive also and that time may be ahead of the linear period, in order to compensate for the transmission delay, if the goal is the server-side to display or do something with the image at actual frame time.

I think you have to synchronize/measure the difference of the clocks of the client and the server with an initial hand-shake or as part of the transactions, each time, and to include and log the time of sending/receiving in the transactions/log.

That way you could measure some average delay of the transmission.

Another thing that may give hints is initially to play the images without sending them. That will show how much time cv2.imread(...) and the preprocessing take.

I.e. commenting/adding another function without destination["currentSocket"].send(jpg_as_text)