PubNub to server data transfer

Question

I am building an IoT application. I am using PubNub to communicate between the harware and the user. Now I need to store all the messages and data coming from the hardware and from the user in a central server. We want to do a bit of machine learning.

Is there a way to do this other than having the server subscribe to all the output channels (There will be a LOT of them)?

I was hoping for some kind of once-a-day data dump involving the storage and playback module in PubNub

Thanks in advance

Answer 1

PubNub to Server Data Transfer

Yes you can perform once-a-day data dumps involving the storage and playback feature.

But first check this out! You can subscribe to Wildcard Channels like a.* and a.b.* to receive all messages in the hierarchy below. That way you can receive messages on all channels if you prefix each channel with a root channel like: root.chan_1 and root.chan_2. Now you can subscribe to root.* and receive all messages in the root.

To enable once-a-day data dumps involving Storage and Playback, first enable Storage and Playback on your account. PubNub will store all your messages on disk over multiple data centers for reliability and read latency performance boost. Lastly you can use the History API on your server to fetch all data stored as far back as forever as long as you know the channels to fetch.

Here is a JavaScript function that will fetch all messages from a channel.

Get All Messages Usage

get_all_history({
    limit    : 1000,
    channel  : "my_channel_here",
    error    : function(e) { },
    callback : function(messages) {
        console.log(messages);
    }
});

Get All Messages Code

function get_all_history(args) {
    var channel  = args['channel']
    ,   callback = args['callback']
    ,   limit    = +args['limit'] || 5000
    ,   start    = 0
    ,   count    = 100
    ,   history  = []
    ,   params   = {
            channel  : channel,
            count    : count,
            callback : function(messages) {
                var msgs = messages[0];
                start = messages[1];
                params.start = start;
                PUBNUB.each( msgs.reverse(), function(m) {history.push(m)} );
                callback(history);

                if (history.length >= limit) return;
                if (msgs.length < count)     return;

                count = 100;
                add_messages();
            },
            error : function(e) {
                log(  message_out, [ 'HISTORY ERROR', e ], '#FF2262' );
            }
        };

    add_messages();
    function add_messages() { pubnub.history(params) }
}