How to Log to a variable or write observer that sends messages to variable in Twisted / Autobahn

Question

I am writing a websocket client that will receive updates every few seconds or so utilizing autobahn with twisted. I am successfully logging the data using multiple observers, however I want to use part of the messages I am receiving to send to a dataframe (and eventually plot in real time). My assumption is that I can log to a variable as well as a file-like object, but I cannot figure out how to do that. What is the correct way to achieve this.

I have very thoroughly read the docs for the current and legacy twisted loggers:

twisted.log https://twistedmatrix.com/documents/current/core/howto/logging.html
twisted.logger https://twistedmatrix.com/documents/current/core/howto/logger.html

In my code I have tried to use a zope.interface and @provider as referenced in the new twisted.logger package to create a custom log observer but have had no luck thus far even getting a custom log observer to print, let alone even send data to a variable.

from twisted.internet import reactor
from autobahn.twisted.websocket import WebSocketClientFactory, WebSocketClientProtocol, connectWS
from twisted.logger import (globalLogBeginner, Logger, globalLogPublisher,
  jsonFileLogObserver, ILogObserver)

import sys
import io
import json

from pandas import DataFrame

def loggit(message):
    log.info("Echo: {message!r}", message=message)



class ClientProtocol(WebSocketClientProtocol):
    def onConnect(self, response):
        print("Server connected: {0}".format(response.peer))

    def initMessage(self):
        message_data = {}
        message_json = json.dumps(message_data)
        print "sendMessage: " + message_json
        self.sendMessage(message_json)

    def onOpen(self):
        print "onOpen calls initMessage()"
        self.initMessage()

    def onMessage(self, msg, binary, df):
        loggit(msg)


    def onClose(self, wasClean, code, reason):
        print("WebSocket connection closed: {0}".format(reason))


if __name__ == '__main__':

    factory = WebSocketClientFactory("wss://ws-feed.whatever.com")
    factory.protocol = ClientProtocol

    @provider(ILogObserver)
    def customObserver(whatgoeshere?):
        print event

    observers = [jsonFileLogObserver(io.open("loga.json", "a")),
     jsonFileLogObserver(io.open("logb.json", "a")), customObserver(Whatgoeshere?)]

    log = Logger()

    globalLogBeginner.beginLoggingTo(observers)



    connectWS(factory)
    reactor.run()

Answer 1

A log observer is simply a callable object that takes a dictionary containing all the values that are part of the log message.

This means you can have an instance of a class with a __call__ method decorated with @zope.interface.implementer(ILogObserver), or a function decorated with @zope.interface.provider(ILogObserver), which can perform that role.

Here's an example of some code which logs some values to a text file, a JSON file, and an in-memory statistics collector which sums things up on the fly.

import io
from zope.interface import implementer
from twisted.logger import (globalLogBeginner, Logger, jsonFileLogObserver,
                            ILogObserver, textFileLogObserver)

class Something(object):
    log = Logger()

    def doSomething(self, value):
        self.log.info("Doing something to {value}",
                      value=value)

@implementer(ILogObserver)
class RealTimeStatistics(object):
    def __init__(self):
        self.stats = []

    def __call__(self, event):
        if 'value' in event:
            self.stats.append(event['value'])

    def reportCurrent(self):
        print("Current Sum Is: " + repr(sum(self.stats)))

if __name__ == "__main__":
    stats = RealTimeStatistics()
    globalLogBeginner.beginLoggingTo([
        jsonFileLogObserver(io.open("log1.json", "ab")),
        textFileLogObserver(io.open("log2.txt", "ab")),
        stats, # here we pass our log observer
    ], redirectStandardIO=False)
    something = Something()
    something.doSomething(1)
    something.doSomething(2)
    something.doSomething(3)
    stats.reportCurrent()