Decoding ROS messages from ROSBridge

Question

How can I decode rosbridge data in the browser?

So far I've been able to decode the following types:

• Uncompressed raw RGB
• Uncompressed raw Depth
• JPEG compressed RGB

My Problem now is decoding compressed depth and PointCloud2 data. As far as my understanding goes, the data is encoded as base64. The depth image has been compressed to a mono16 PNG. I have tried many different approaches, but none seem to work. The depth image is supposed to contain 307200 depth values of 16 bits each.

I do not want to display this data in something like ros3djs or webviz (cloud not figure out how they do the decoding). I want to decode the data and use it in my own analysis.

Steps to reproduce:

Here is a sample file. It contains the data field of the JSON message: https://drive.google.com/file/d/18ZPpWrH9TKtPBbevfGdceZVpmmkiP4bh/view?usp=sharing

OR

1. Make sure you have a device publising or a rosbag playing
2. roslaunch rosbridge_server rosbridge_websocket.launch
3. Launch your web page and make sure you added roslibjs in your script tag

The JS of my web page is simplified to this:

var ros = new ROSLIB.Ros({
    url: 'ws://127.0.0.1:9090'
});


var depthListener = new ROSLIB.Topic({
    ros: ros,
    name: '/camera/color/image_raw/compressedDepth',
    messageType: 'sensor_msgs/CompressedImage'
});

var pointCloudListener = new ROSLIB.Topic({
    ros: ros,
    name: '/camera/depth/color/points',
    messageType: 'sensor_msgs/PointCloud2'
});



depthListener.subscribe(function (message) {
    console.log(message);
    depthListener.unsubscribe();
});

pointCloudListener.subscribe(function (message) {
    console.log(message);
    pointCloudListener.unsubscribe();
});

I have set the two topics to unsubscribe after the first message so that my console does net get flooded.

Provided a screenshot of the console logs for depth image

and for pointcloud

This is what I have so far, but the onload function is never triggered.

image = new Image();
    image.src = "data:image/png;base64, " + message.data

    image.onload = function(){
        image.decode().then(() =>{
            if(image.width != 0 && image.height != 0){
                canvas.width = image.width;
                canvas.height = image.height;
                ctx = canvas.getContext('2d');
                ctx.drawImage(image, 0, 0);
                image_data = canvas.getContext('2d').getImageData(0,0, 640,480).data;
            }
        });
    }

I think this OpenCV code was used to compress the image. The message can essentially be thought of as a 16 bit gray scale image.

Please comment if I can update the question with specific information

Answer 1

According to libpng a PNG starting signature is

 89  50  4e  47  0d  0a  1a  0a

As pointed here, the signature is present after a header (few initial 00 bytes in your case). This would solve your problem:

function extractPng(base64) {
  // const signature = '\x89\x50\x4e\x47\x0d\x0a\x1a\x0a';
  const signature = '\x89PNG\r\n\x1a\n';
  let binary = window.atob(base64);
  let ix = binary.indexOf(signature);
  ix = ix > 0 ? ix : 0;
  return 'data:image/png;base64,' + window.btoa(binary.substring(ix));
}

Your image

[640 x 480]

As you can see there are some very dark gray areas

Contrasted version

Complete Example

This is a complete example that displays you the image too:

function openDataImage(data) {
  const image = new Image();
  image.src = data;
  const w = window.open("");
  w.document.write(image.outerHTML);
}

openDataImage(extractPng(`...`));

Complete Decoding

Unfortunately <canvas> has a 8bit fixed color depth, hence it can not be used to access your 16bit grayscale data. I suggest using pngjs. Pngjs is not available (at least I have not found it) as a compiled browser-ready library so you will need to package your 'website' somehow (like with Webpack).

The function will need to extract the png binary data as a Buffer:

function extractPngBinary(base64) {
  const signature = Buffer.from("\x89PNG\r\n\x1a\n", "ascii");
  let binary = Buffer.from(base64, "base64");
  let ix = binary.indexOf(signature);
  ix = ix > 0 ? ix : 0;
  return binary.slice(ix);
}

Then to decode the png:

const PNG = require("pngjs").PNG;
const png = PNG.sync.read(extractPngBinary(require("./img.b64")));

To read values out of the PNG then (encoding is BE):

function getValueAt(png, x, y) {
  // Check is Monotchrome 16bit
  if (png.depth !== 16 || png.color || png.alpha) throw "Wrong PNG color profile";
  // Check position
  if (x < 0 || x > png.width || y < 0 || y > png.height) return undefined;
  // Read value and scale to [0...1]
  return (
    png.data.readUInt16BE((y * png.width + x) * (png.depth / 8)) /
    2 ** png.depth
  );
}

To read a region of data then:

function getRegion(png, x1, x2, y1, y2) {
  const out = [];
  for (let y = y1; y < y2; ++y) {
    const row = [];
    out.push(row);
    for (let x = x1; x < x2; ++x) {
      row.push(getValueAt(png, x, y));
    }
  }
  return out;
}

All the image:

getRegion(png, 0, png.width, 0, png.height);

Here a complete example (with source code)

The button "Decode Image" will decode the depths of a small area.