Search code examples
openwhiskrhea

OpenWhisk and binary data from Google Flatbuffers

We have data being created by a simulated device being put on the network with NanoMSG with a payload of Google FlatBuffers (binary).

We would like to trigger on patterns of this data with OpenWhisk, and respond with Flatbuffer encoded responses.

Assume latency and throughput are not a big concern here.

Which approach can we take:

  1. Write a repeater that converts the Flatbuffer to JSON (FB has a utility to do this) and then place the data onto an AMQP buss which is listened to by OpenWhisk? (we have folks familiar with AMQP, but not Kafka

  2. Try to do something with Kafka, which seems (maybe it is only the IBM version) to directly handle the binary Flabuffers (probably still need a shim from NanoMSG to Kafka. E.g.

How to invoke an OpenWhisk action from IoT Platform in Bluemix

https://medium.com/openwhisk/serverless-transformation-of-iot-data-in-motion-with-openwhisk-272e36117d6c

Not sure if we still don't need the Flatbuffers JavaScript deserializer and serializer to convert the binary based64 data in JavaScript to JSON

  1. Learn Kakfa, and then transform the NanoMsg payload (Flatbuffers to JSON).

  2. Something else?

Anyone have direct experience in this?

Update

Thank you James, those are spot-on links. But it does raise some secondary issues:

  1. If the data is in Google FlatBuffers schema, it does not seem to be any advantage to using Kafka binary transformation, since the mux/demux from base64 still needs to be done in the javascript layer.
  2. It is slightly disturbing that Kafka (which is known for its low latency) is batching the events. That does effect latency when one has Iot (sensor data) that needs to be responded to in a closed-loop to actuators (sensor->control->actuators) is a common robotics model, and that is pretty much close to what we are doing. For the moment we are not pushing the latency issue, but I can see emerging cases where we will need the low latency. What is the thinking in the Kafka Whisk provider community about this?
  3. I must be missing something, but the AMQP provider says it is using RHEA https://github.com/amqp/rhea#receiver . That seems to provide all one needs in terms of writing a simple trigger/rules for dealing with sensor stream data. Why would one use OpenWhisk?
Solution

  • Either option makes sense. OpenWhisk actions receive and return JSON messages. Binary data passed into those functions must be Base64 encoded.

    If you use an AMQP feed, you can convert the binary data to JSON manually.

    The Kafka feed provider does support automatic encoding of the binary input values (using the isBinary* parameters).

    Kafka feeds push batches of messages to the OpenWhisk actions. This is different from a message queue, which would push one message at a time. This feed provider is built-in OpenWhisk.

    There is an external community feed provider for AMQP here. This would need you to install and run it manually.