What is the best way to subscribe and publish to multiple topics in paho-mqtt?

Question

I'm using MQTT to disturb messages in my network and have an question about the cleanest way to publish and subscribe multiple messages to the broker.
First of all, I've got two lists:

request_list = [('sensors/system/temperature', 0), 
                ('sensors/system/gyroscope', 1), 
                ('sensors/system/acceleration', 2)]

Which contains my topics I have to publish my messages to.
My second list defines the messages I want to publish and the topics where I want to get my response (== the topics I have to subscribe to get my answers).

request_value = ['{"response":"similarity/sensors/system/temperature","duration":"60s"}',
                  {"response":"similarity/sensors/system/gyroscope","duration":"60s"}', 
                 '{"response":"similarity/sensors/system/acceleration","duration":"60s"}'] 

My broker is for every topic the same and defined with HOST= "192.168.137.1" on PORT = "8083".
For now I'am using a for loop, to subscribe to one topic, publish my message and wait for the message to come in. Because I have to wait for every subscribtions and publish to suceed its very time consuming. The pseudocode of my current code looks like the following:

list_measurments = []
for topic in request_list:
    client.connect("Broker","Host")
    client.loop_start() 
    client.subscribe("sub_topic")
    client.pub("pub_topic","pub_message")
    client.callback("append list_measurements")
    client.loop_stop() #stop the loop
    client.disconnect

I tried to use threads form my question here but it turned out that the normal use of threads would be to publish the same message to a lot of different brokers. I also thought about multiple subscribtions.
If anybody could give me an hint, what the cleanest and fastest approach would be, I'd be very thankful.

Answer 1

You should only connect to the broker and start the client loop once outside the for loop.

Setting up and tearing down the connection to the broker every time will add a huge amount of overhead and leaves lots of room to miss messages.

You should also just subscribe to all the topics you want once at startup. OK, you can add more or unsubscribe if needed, but if they are always the same just subscribe when you connect.

The basic general approach should look like this.

def on_connect(client, userdata, flags, rc):
  for i in request_value:
    client.subscribe(i.response)

  for i in request_list:
    # this loop needs to be a bit more complicated
    # as you need to pull the topic from the request_list
    # and the payload from request_value
    client.publish(i)

def on_message(client, userdata, message):
  if message.topic == "similarity/sensors/system/temperature":
    # update temperature
  elif message.topic == "similarity/sensors/system/gyroscope":
    # update gyro
  elif message.topic == "similarity/sensors/system/acceleration":
    # update accel

client.on_connect = on_connect
client.on_message = on_message
client.connect("BrokerIP")
client.loop_start()

You can also run the publish loop again if needed (as it looks like you are only requesting 60s of data at a time). You would probably do better to combine the request_list and request_value data structures into one list.