Android camera image transport to ROS in real-time have an OOM trouble?

I'm trying to transport the image-info from Android Camera to ROS in real-time. However, I got a OOM problem. I'm new to Android-ROS, nearly have no experiences of dealing with such problem.

Here're some information of my demo: (if you guys need more, pls comment)

public class MainActivity extends RosActivity implements NodeMain, SurfaceHolder.Callback, Camera.PreviewCallback

2.Dependencies Opencv-for-Android(3.2.0).

3.ROS messages type: android_cv_bridge.

I'm trying to publish the image-messages in onPreviewFrame() function. Code like this:

@Override
public void onPreviewFrame(byte[] data, Camera camera) {
    Camera.Size size = camera.getParameters().getPreviewSize();
    YuvImage yuvImage = new YuvImage(data, ImageFormat.NV21, size.width, size.height, null);
    Bitmap bmp = null;

    if(yuvImage != null){
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        yuvImage.compressToJpeg(new Rect(0, 0, size.width, size.height), 80, baos);
        bmp = BitmapFactory.decodeByteArray(baos.toByteArray(), 0, baos.size());

        try{
            baos.flush();
            baos.close();
        }
        catch(IOException e){
            e.printStackTrace();
        }

        image = imagePublisher.newMessage();
        Time curTime = connectedNode.getCurrentTime();

        image.setEncoding("rgba8");
        image.getHeader().setStamp(curTime);
        image.getHeader().setFrameId("camera");

        curTime = null;

        if(isOpenCVInit){
            Mat mat_image = new Mat(bmp.getHeight(), bmp.getWidth(), CvType.CV_8UC4, new Scalar(0));
            Bitmap copyBmp = bmp.copy(Bitmap.Config.ARGB_8888, true);

            // bitmap to mat
            Utils.bitmapToMat(copyBmp, mat_image);
            // mat to cvImage
            CvImage cvImage = new CvImage(image.getHeader(), "rgba8", mat_image);

            try {
                imagePublisher.publish(cvImage.toImageMsg(image));
            } catch (IOException e) {
                e.printStackTrace();
            }

            mat_image.release();
            mat_image = null;

            if(!bmp.isRecycled()) {
                bmp.recycle();
                bmp = null;
            }
            if(!copyBmp.isRecycled()) {
                copyBmp.recycle();
                copyBmp = null;
            }
            cvImage =null;
            image = null;
        }
    }

    yuvImage = null;
    System.gc();
}

The imagePublisher are initialized here:

@Override
public void onStart(ConnectedNode connectedNode) {
    this.connectedNode = connectedNode;
    imagePublisher = connectedNode.newPublisher(topic_name, sensor_msgs.Image._TYPE);
}

Well, I had try my best to avoid the OOM problem. I had also trying to not apply the OpenCV, and just dealing with the bitmap like this:

ChannelBufferOutputStream cbos = new ChannelBufferOutputStream(MessageBuffers.dynamicBuffer());

bmp.compress(Bitmap.CompressFormat.JPEG, 80, baos);
cbos.buffer().writeBytes(baos.toByteArray());
image.setData(cbos.buffer().copy());

cbos.buffer().clear();
imagePublisher.publish(image);

Unfortunately, it's get worse. I'm doubt the way I'm trying to achieve this target. Or is there a better way to do?

Solution

I think your problem might be that your network can't transfer this amount of image data and the OOM is caused by the data stuck in buffers that is not yet transferred.

I had similar issues when I wanted to transfer image from my android device. If your problem is the same, you could solve it in several ways:

transfer data via usb tethering, it's generally much faster than wifi or cellular and can transfer even raw image stream without compression with 30 fps 640x480. For Jpeg I think you will be able to stream FullHD at 30 fps.
save data on the phone to a ROS Bag http://wiki.ros.org/rosbag and then work with the data. Here you miss realtime, but sometimes it's not needed. To make it I actually wrote an application for android https://github.com/lamerman/ros_android_bag and you can also download it directly from Google Play https://play.google.com/store/apps/details?id=org.lamerman.rosandroidbag&hl=en
try to decrease the bandwidth even further (decrease image size, fps) or increase the network quality

About your second attempt with transferring JPEG instead of RAW data, have a look at this source code, here it's implemented correctly https://github.com/rosjava/android_core/blob/kinetic/android_10/src/org/ros/android/view/camera/CompressedImagePublisher.java#L80

The problem of transferring via network is for sure actual for raw images, but may also be for compressed ones if the size of image is big and the frame rate is high.