Search code examples
c++opencvpixelformatopencv-matbasler

Basler Pylon C API and Pixel Format Selection

Intent

I want to develop a machine vision application for high performance applications. Hence I wish to minimise the amount of image processing performed later on after image capture. I require colour as it will be a detection parameter later on, grey scale will not be applicable. I am capturing images as simultaneously as possible given current software trigger through two USB3 cables.

I am using OpenCV and the OpenCV Hardware Accelerated Module (CUDA). I want to transfer images from the cameras into an OpenCV Mat and OpenCV GpuMat. I am also utilising the Basler C++ API for interaction with the cameras.

Hence Two Parameters in play given the situations.

  1. The Mat Type CV_8CU3 ect...
  2. The Pixel Format on camera BGR8 ect...

Hardware

Basler acA2040-55uc USB3 x2 cameras

Pixel Format

As per the Basler website : acA2040-55uc - Basler ace features

BGR 8

Bayer RG 12

Bayer RG 12p

Bayer RG 8

Mono 8

RGB 8

YCbCr422_8 (YUV422_8)

Issues

  1. Via the pylon viewer the pixel format parameter is greyed out. I.e the option is unmodifiable and stuck as Bayer RG 8. This is confusing given the information provided on the website.

  2. In C++, using the Pylon API. The persistent issue of formatting as been playing up. I cannot get access to all the stated pixel formats. I.e. a 3 channel cv::Mat + BGR8 does not work, in fact the Basler API specifies the error that the pixel format is incompatible given a selection of BGR8. I have only ever got this to work with a 2 channel cv::Mat for Bayer RG 8 which seems to go against all the Basler stated documentation.

Resolutions

I would like to understand what compatible pixel formats I can use, and to know other peoples experience with this platform.

Additional question, if someone can explain to me how pixel format plays into computer vision and performance application that would be great. I can only ever find definitional information regarding pixel formats and have no practical understanding of how this applies to machine vision bar the fact they reside in different colour spaces.

Solution
    1. Image Format parameter is not available when camera is in Continuous Grab mode (StartAcquisition API command successfully issued). Stop acquisition in pylon Viewer and then the parameter should be enabled in parameter tree. If not, contact Basler support, because it would be unusual behavior for camera.
    2. Paste your conversion code pls.

    Pixel Format parameter determines how intensity information is converted into byte buffer in terms of alignment of channel data, number of bytes etc. 'Bayer..' formats are usually a raw output from color sensor having a Bayer mask on its pixels. As this is actual data coming physically to camera's FPGA, every other pixel format is somehow converted by FPGA on-fly by process of so called debayerization. BGR/RGB formats are 3 byte per pixel formats - by setting them up in camera, you naturally slow down the acquisition, because for every raw frame camera produces/reconstructs 3 channels. Mono image on color camera is also computed by special weighing algorithm. There are also YUV formats that convert RGB information to more human-specific color space (chroma/luma ect) that can assign lower number of bits to channel that is less sensitive by human eye - fancy math behind, wiki for further reading.

    On top of above which are mainly 8bit color formats, you have additional Bayer12 formats that transfer native data of 12 bits per channel, as spit out by sensor's ADC, resulting in 4096 grey levels. You have then further choice of Bayer 12 (12 bit image info in 16 bits of data, 4 bits unused), or Bayer 12p (12 bit image info in 12 bits of data, next pixel starts in padded 4 bits - 2 pixels coded in 3 bytes). Rarely used though, my record only with some special digitization/archiving projects. Vast majority of applications suffices 8 bit with no issues.

    Though I'm not an OpenCV nerd at all, I believe once you grasp Basler's image formats (btw they are now standarized by GenICam's PFNC specification), you'll be able to find a corresponding image format for OpenCV. I can suggest:

    1. transferring YUV422 format from camera to memory buffer, using Basler bundled Converter class and then feeding it to CV::Mat class. This way you maintain camera's original framerate for color image data.
    2. Alternatively you can use Bayer RG 8 and use proper debayerization algorithm on computer side. If framerate is of no importance, you can surely switch camera to RGB/BGR 3bytes/pixel formats, at bandwidth/latency cost.

    Furter reading: https://docs.baslerweb.com/pixel-format