Which real-time communication protocol between hardware and web components?

Question

I wasn't entirely sure how to word my question in the title so apologies if it's confusing.

I'd like to build a system that would function as a sort of information dashboard for my home. It would consist of a number of hardware and software components that would ultimately result in a simple, clean website with real-time displays of a number of analog sensors such as temperature, wind speed and direction, etc.

I've got a good idea of what I'm going to do for the hardware, as well as for displaying the information; my question has to do with the communication between the hardware and web server.

I'd like the hardware to fire messages at a fairly fast rate so I don't think HTTP POST will suffice. I'm also not extremely concerned with receiving 100% of the messages but receiving as many as possible is definitely a plus. The data will be coming from the hardware, populating some sort of database (likely Redis).

So far, I've researched a couple of things but I'm not sure I'm heading in the right direction. I've looked in to message-oriented middleware such as RabbitMQ but I'm not convinced I need the overhead. I've also looked into Redis Pub/Sub which seems like a more appropriate solution since I'd like the web app to chart out say the last 5 minutes of data but even then I'm not certain. Can I just fire UDP packets to a custom-built listener?

I'm pretty certain the hardware will be two stages (a uC feeding a small embedded linux machine) so you might even liken this to desktop software firing messages to a web server as quickly as possible.

I'm venturing into an area that I know absolutely nothing about so any guidance is much appreciated.

Answer 1

Like the other poster mentioned, your're not going to have issues with http post. Node's http implementation is built for high concurrency.

Personally, I think I'd go with:

1. Hardware device output ->
2. Linux box fires an http post directly to your central server (node.js) ->
3. Take your changes and immediately publish them to your web client via socket.io (real time transport for the browser). https://github.com/LearnBoost/Socket.IO/

Socket.io is probably the best out of the box real time transport for node.js <==> browser

If you want persistence, redis isn't bad (plus you get the free pub/sub) if your data fits into that model, which it probably does.

There's no reason you couldn't use a reglar tcp based connection also (net module) if thats your bag. Unless you want the data to be unreliable, I wouldn't go to udp. Thats more lossy streaming media oriented.

I really doubt that you're going to have enough messages to worry about a dedicated message queue. Rabbitmq introduces features like queue persistence and is built for incredibly high throughput. Probably orders of magnitude more than what you're after.

You might look at a library like zeromq for which there are node bindings: https://github.com/JustinTulloss/zeromq.node. Its like a topic/other type of exchange in rabbitmq, but without a central message queue node (one calls this brokerless). That way you can just talk directly to your linux/hardware nodes, but still get the message queue like interface -- you publish your hardware updates on a 'channel' and your server nodes listens for such updates.