Streaming video of wifi access point camera to a remote computer

Question

After spending weeks of searching the forums and trying different approaches, I didn't find a solution for my quite specific problem. I'm thankful for every hint you might provide.

I purchased a Kodak Pixpro 360 camera which offers a view-finder function over wifi (i.e. a live video stream). Now I'm trying to use this camera as a surveillance cam that can be accessed from anywhere (not just the local network). An ODROID will be connected to the camera via wifi and use a second wifi dongle to connect to the LAN. The incoming video stream should be forwarded in real-time to the client (there will be only one at a time). The received 360 degree content is then viewed in an application written in Unity3d.

So far I have managed to grab the cam's MJPEG stream and serve it as JPEGs unsing a NodeJS server. The JPEGs are then rendered via the WWW.LoadImageIntoTexture method. As you might imagine, GET requests for each frame are horribly slow and result in about 0.5 frames per second.

A colleague of mine pointed me to WebRTC and the Janus gateway as a more elegant solution. This simple peer chat uses SocketIO and is working just fine with my webcam but I can not figure out how to change this code to use the video stream coming from the PIXPRO instead of my local device. Rendering the content should be fun, too, as you need a browser for WebRTC and I am not sure how much of that can be embedded in Unity3d.

Unfortunatelly, the camera can not connect to a LAN by itself but rather acts as a wifi access point. This makes all the solutions I found for ip cams obsolete.

I found a similar project that managed to forward their video stream via Janus and WebRTC but I am not sure if and how I can apply their methods. https://babyis60.wordpress.com/2015/02/04/the-jumping-janus/

UPDATE

Ok guys, I managed to narrow down my problem by myself. The PIXPRO has no RTP support, so I am stuck with the JPEG Stream. Now I am trying to speed up the paparazzo.js implementation that reads the camera's TCP responses and returns JPEGs by searching for the boundary between the frames. These JPEGs are then served via a http response. I would like to speed up this process by using SocketIO to push these frames to the client and render them there. The strange thing is that the data seems to be just fine on serve side (I get a valid JPEG image when I export it via fs.writeFileSync('testimage.jpg', buffer, 'binary');, but I can't get it to work on client side after I send the image via io.sockets.emit("stream",{image: image});. When I try to display this image in a browser via $("#video").attr("src", "data:image/jpeg;," + data.image);, the image is not parsed corretly. The inspector shows that the video source is updated, but there is only a binary string.

Answer 1

I finally managed to get it done. The binary had to be loaded into a Buffer and sent as a base64 string.

paparazzo.on("update", (function(_this) {
    return function(image) {
      updatedImage = image;

      var vals = new Buffer(image, 'binary');
      //fs.writeFileSync('testimage.jpg', vals, 'binary');
      io.sockets.emit("image",vals.toString('base64'));
      return console.log("Downloaded " + image.length + " bytes");
    };
  })(this));

On client side I had to use an image tag because canvas solutions didn't seem to work for me.

var image = document.getElementById('image');

socket.on("image", function(info) {

    image.src = 'data:image/jpeg;base64,' + info;

});

The browser output was just a test before the actual Unity3D implementation. I tried many Websocket libraries for Unity3D, but the only one that worked on an Android device was the UnitySocketIO-WebsocketSharp project. Now I could simply convert my base64 image to a byte array and load it into a Texture2D.

socket.On("image", (data) => {
        bytes = Convert.FromBase64String (data.Json.args.GetValue(0).ToString());
});

void Update () {
    tex.LoadImage (bytes);
}

The LoadImage seems to block the UI threat, though, which slows down my camera control script so I will have to take a look into Unity plugins that can re-write texture pixels on a lower level. Using the Cardboard SDK for Unity worked a work-around for me to get a quite smooth camera control again.