Im currently working on a project that involves a number of small linux based hosts which read data from a sensor and send the data through an Open VPN link to a server. The server is an Ubuntu VM which will be able to scale to cope with demand as the network grows.
Initially we will have 20 sensors distributed around the campus however the end solution may need to scale into the thousands.
Each sensor will pass data up to the server every 15 minutes. The data size per transaction will be of the order of 100 bytes.
The data will be placed inside a round robin database on the server and ultimately made available for review through an apache web interface.
I want to know what the best solution which is currently available using open source technology is to achieve this transaction from sensor to server.
We are considering ssh, scp, netcat and web based solutions however alternatives will be considered.
The common implementation for this sort of usage is a messaging protocol; the most popular one for sensors, IoT etc. is MQTT. There are numerous open source libraries in pretty much every language you want, and several open source brokers, e.g. MosQuitto.
The technologies you mention look like they are synchronous - "sensor opens SSH shell on server and executes BASH command", "sensor executes HTTP request on server and PUTS data". This is the equivalent of contacting someone by phone - if they're not there, you have to wait while the phone rings, and your message may get lost (let's ignore voice mail for the sake of this example).
This is problematic in most scenarios - sensors are usually connected using unreliable protocols, and have limited CPU and storage capacity. A messaging-based, asynchronous protocol requires the sensor to send a message with the data, and get on with its life; the messaging solution deals with error handling, missing connectivity, etc. This is the equivalent of sending an email - even if you're not on the network when you send it, or the recipient is away, eventually the message will get there.
MQTT is a really lightweight and flexible messaging solution, designed for clients with limited hardware.