proper threading in python

Question

I am writing a home automation helpers - they are basically small daemon-like python applications. They can run each as a separate process but since there will be made I decided that I will put up a small dispatcher that will spawn each of the daemons in their own threads and be able to act shall a thread die in the future.

This is what it looks like (working with two classes):

from daemons import mosquitto_daemon, gtalk_daemon
from threading import Thread

print('Starting daemons')
mq_client = mosquitto_daemon.Client()
gt_client = gtalk_daemon.Client()

print('Starting MQ')
mq = Thread(target=mq_client.run)
mq.start()

print('Starting GT')
gt = Thread(target=gt_client.run)
gt.start()

while mq.isAlive() and gt.isAlive():
    pass
print('something died')

The problem is that MQ daemon (moquitto) will work fine shall I run it directly:

mq_client = mosquitto_daemon.Client()
mq_client.run()

It will start and hang in there listening to all the messages that hit relevant topics - exactly what I'm looking for.

However, run within the dispatcher makes it act weirdly - it will receive a single message and then stop acting yet the thread is reported to be alive. Given it works fine without the threading woodoo I'm assuming I'm doing something wrong in the dispatcher.

I'm quoting the MQ client code just in case:

import mosquitto
import config
import sys
import logging


class Client():
    mc = None

    def __init__(self):
        logging.basicConfig(format=u'%(filename)s:%(lineno)d %(levelname)-8s [%(asctime)s]  %(message)s', level=logging.DEBUG)
        logging.debug('Class initialization...')
        if not Client.mc:
            logging.info('Creating an instance of MQ client...')
            try:
                Client.mc = mosquitto.Mosquitto(config.DEVICE_NAME)
                Client.mc.connect(host=config.MQ_BROKER_ADDRESS)
                logging.debug('Successfully created MQ client...')
                logging.debug('Subscribing to topics...')
                for topic in config.MQ_TOPICS:
                    result, some_number = Client.mc.subscribe(topic, 0)
                    if result == 0:
                        logging.debug('Subscription to topic "%s" successful' % topic)
                    else:
                        logging.error('Failed to subscribe to topic "%s": %s' % (topic, result))
                logging.debug('Settings up callbacks...')
                self.mc.on_message = self.on_message
                logging.info('Finished initialization')
            except Exception as e:
                logging.critical('Failed to complete creating MQ client: %s' % e.message)
                self.mc = None
        else:
            logging.critical('Instance of MQ Client exists - passing...')
            sys.exit(status=1)

    def run(self):
        self.mc.loop_forever()

    def on_message(self, mosq, obj, msg):
        print('meesage!!111')
        logging.info('Message received on topic %s: %s' % (msg.topic, msg.payload))

Answer 1

You are passing Thread another class instance's run method... It doesn't really know what to do with it.

threading.Thread can be used in two general ways: spawn a Thread wrapped independent function, or as a base class for a class with a run method. In your case it appears like baseclass is the way to go, since your Client class has a run method.

Replace the following in your MQ class and it should work:

from threading import Thread

class Client(Thread):
    mc = None

    def __init__(self):
        Thread.__init__(self) # initialize the Thread instance
        ...
    ...

    def stop(self):
        # some sort of command to stop mc
        self.mc.stop() # not sure what the actual command is, if one exists at all...

Then when calling it, do it without Thread:

mq_client = mosquitto_daemon.Client()
mq_client.start() 

print 'Print this line to be sure we get here after starting the thread loop...'