Search code examples
c++cgoogle-project-tango

Why does the image extracted from the Project Tango Tablet appear gray?


  • I am using the Project Tango C API
  • I am writing the entire buffer pointed to by TangoImageBuffer.data to a file.

    // open a file
    std::fstream f(app.getNextFileName(), std::ios::out | std::ios::binary);
    
    // write the entire stride * height * 4 buffer into the file
    f.write((const char *) tango_image_buffer->data, 
        tango_image_buffer->stride * buffer->height * 4 * sizeof(uint8_t)
    );
    
    f.close();
    
  • I then export the file to my PC and visualize it using python-opencv and numpy:

    import sys
    
    import cv2
    import numpy as np
    
    # dimensions of the image
    stride    = 1280
    width     = 1280
    height    = 720
    
    # I am using 3 channels so that the resulting image is not
    # transluscent
    channels  = 3
    
    input_filename  = sys.argv[1]
    output_filename = sys.argv[2]
    
    # load the image buffer into a list
    data = np.fromfile(input_filename, dtype=np.uint8)
    
    # create a height x width x channels matrix with the datatype uint8 
    # and all elements set to zero
    img = np.zeros((height, width, channels), dtype=np.uint8);
    
    # map elements in array to image matrix
    for y in range(0, height):    
        for x in range(0, width):
            img[y, x, 0] = data[y * stride + x + 2] #blue
            img[y, x, 1] = data[y * stride + x + 1] #green
            img[y, x, 2] = data[y * stride + x + 0] #red
    
    # display and save the resulting image
    cv2.namedWindow("tango-rgba")
    cv2.imshow("tango-rgba", img)
    cv2.waitKey(0)
    cv2.destroyAllWindows()
    cv2.imwrite(output_filename, img)
    
  • Unfortunately, the resulting image looks gray

  • The image only appears to be gray. Upon closer inspection of the pixels, you will find that the red, green, and blue components of the pixels are different.

  • EDIT: I have changed this post to specify that the buffer I am using is in RGBA8888 format. From the Project Tango Reference, "Only RGBA 8888 is provided"

  • EDIT: It actually seems RGBA8888 format is not used. rhashimoto suggests that the format is YV12.

  • EDIT: The image format is actually NV21 (See answer below).


Solution

  • It seems that as of writing this post, the Project Tango Reference really is incorrect.

    The image format is NV21.

    char format_str[100] = {0};
    
    switch (tango_image_buffer->format)
    {
      case TANGO_HAL_PIXEL_FORMAT_RGBA_8888:
            sprintf(format_str, "%s", "RGBA8888");
            break;
      case TANGO_HAL_PIXEL_FORMAT_YV12:
            sprintf(format_str, "%s", "YV12");
            break;
      case TANGO_HAL_PIXEL_FORMAT_YCrCb_420_SP:
            sprintf(format_str, "%s", "NV21");
            break;
      default:
            break;
    
    }
    __android_log_print(ANDROID_LOG_VERBOSE, "Capture Info",
            "stride: %u, width: %u, height: %u, timestamp: %lf, frame: %llu, format: %s",
            tango_image_buffer->stride, 
            tango_image_buffer->width, 
            tango_image_buffer->height,
            tango_image_buffer->timestamp, 
            tango_image_buffer->frame_number, 
            format_str);
    

    The output of the above piece of code is:

       V/Capture Info﹕ stride: 1280, width: 1280, height: 720, timestamp: 22157.368703, frame: 0, format: NV21
    

    Here is my new visualization code in python using opencv and numpy:

    import sys
    
    import cv2
    import numpy as np
    
    input_filename  = sys.argv[1]
    output_filename = sys.argv[2]
    
    
    # dimensions of the image
    stride    = 1280
    width     = 1280
    height    = 720
    channels  = 4
    
    # load file into buffer
    data = np.fromfile(input_filename, dtype=np.uint8)
    
    # create yuv image
    yuv = np.ndarray((height + height / 2, width), dtype=np.uint8, buffer=data)
    
    # create a height x width x channels matrix with the datatype uint8 for rgb image
    img = np.zeros((height, width, channels), dtype=np.uint8);
    
    # convert yuv image to rgb image
    cv2.cvtColor(yuv, cv2.COLOR_YUV2BGRA_NV21, img, channels)
    
    # display and save the resulting image
    cv2.namedWindow("tango-rgba")
    cv2.imshow("tango-rgba", img)
    cv2.waitKey(0)
    cv2.destroyAllWindows()
    cv2.imwrite(output_filename, img)
    

    and the resulting image.

    TLDR: The image format is NV21. OpenCV provides functionality for converting NV21 to RGB. The result looks great.