My issue is a duplicate of this one. There was no solution there. In hopes to find a solution and detail my specific setup, below shows the function used to read frames from the .oni file. If this function is run with Type == 2 (i.e. runs for # of RGBD images, where Criteria is the #), running this function in a for loop should allow a user to access each image. However, the indices for the color image and depth image do not match and are out of order. This continues until the waitForAnyStream times out for all the following calls to IMG_pipeline::listen(...).
void IMG_pipeline::listen(int Type, int Criteria){
int exitNumber;
clock_t start = clock();
double elapsedtime;
openni::VideoFrameRef frame;
int CurrentIMGCount=0;
switch (Type){
case 0:
{
exitNumber = -1;
break;
}
case 1:
{
exitNumber = Criteria;
break;
}
case 2:
{
exitNumber = -1;
break;
}
}
for (int i = 0;i!=exitNumber;i++){
readyStream = -1;
rc = openni::OpenNI::waitForAnyStream(streams, 2, &readyStream, SAMPLE_READ_WAIT_TIMEOUT);
if (rc != openni::STATUS_OK)
{
printf("Wait failed! (timeout is %d ms)\n%s\n", SAMPLE_READ_WAIT_TIMEOUT, openni::OpenNI::getExtendedError());
//break;
}
switch (readyStream)
{
case 0:
{
// Depth
depth.readFrame(&frame);
break;
}
case 1:
{
// Color
color.readFrame(&frame);
break;
}
default:
{
printf("Unexpected stream: %i\n", readyStream);
continue;
}
}
int Height = frame.getHeight();
int Width = frame.getWidth();
cvColor.release();
cvX.release();
cvY.release();
cvZ.release();
cvColor = cv::Mat(Height, Width, CV_8UC3);
cvX = cv::Mat(Height, Width, CV_32F);
cvY = cv::Mat(Height, Width, CV_32F);
cvZ = cv::Mat(Height, Width, CV_32F);
switch (frame.getVideoMode().getPixelFormat())
{
case openni::PIXEL_FORMAT_DEPTH_1_MM:
case openni::PIXEL_FORMAT_DEPTH_100_UM:
{
openni::DepthPixel* pDepth = (openni::DepthPixel*)frame.getData();
int k =0;
for (int ri = 0; ri<Height; ri++)
{
for (int ci = 0; ci<Width; ci++)
{
float pdepth_val = pDepth[k];
openni::CoordinateConverter::convertDepthToWorld(depth, (float)ri, (float)ci, pdepth_val, &cvX.at<float>(ri,ci), &cvY.at<float>(ri,ci), &cvZ.at<float>(ri,ci));
k++;
}
}
TotalFrames[0]++;
XYZCaptured = true;
printf("Frame Index: %i \n", frame.getFrameIndex());
printf("Depth Captured. \n");
break;
}
case openni::PIXEL_FORMAT_RGB888:
{
cvColor.data = (uchar*)frame.getData();
TotalFrames[1]++;
ColorCaptured = true;
printf("Frame Index: %i \n", frame.getFrameIndex());
printf("Color Captured. \n");
break;
}
default:
printf("Unknown format \n");
}
printf("Frame extracted. \n");
if (ColorCaptured && XYZCaptured){
if (NewButNotRead == true){
IMGsMissed++;
}
else
NewButNotRead = true;
ColorCaptured = false;
XYZCaptured = false;
RGBD_out.clear();
RGBD_out.push_back(cvX);
RGBD_out.push_back(cvY);
RGBD_out.push_back(cvZ);
RGBD_out.push_back(cvColor);
CurrentIMGCount++;
printf("Image overwritten. \n");
}
elapsedtime=(clock()-start)/((double)CLOCKS_PER_SEC);
printf("Time since listen initiation: %f \n \n", elapsedtime);
if (CurrentIMGCount ==Criteria && Type == 2)
return;
else if (elapsedtime>(double)Criteria && Type==0)
return;
}
frame.release();
}
Here is a console output example:
Frame Index: 1
Depth Captured.
Frame extracted.
Time since listen initiation: 0.004846
Frame Index: 2
Depth Captured.
Frame extracted.
Time since listen initiation: 0.011601
Frame Index: 1
Color Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.012640
Depth frame count: 3
Color frame count: 2
Frame Index: 54
Color Captured.
Frame extracted.
Time since listen initiation: 0.000067
Frame Index: 57
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.005878
Depth frame count: 4
Color frame count: 3
Frame Index: 96
Color Captured.
Frame extracted.
Time since listen initiation: 0.000079
Frame Index: 99
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.003628
Depth frame count: 5
Color frame count: 4
Frame Index: 126
Color Captured.
Frame extracted.
Time since listen initiation: 0.000048
Frame Index: 130
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.004782
Depth frame count: 6
Color frame count: 5
Frame Index: 152
Color Captured.
Frame extracted.
Time since listen initiation: 0.000065
Frame Index: 156
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.008294
Depth frame count: 7
Color frame count: 6
Frame Index: 181
Color Captured.
Frame extracted.
Time since listen initiation: 0.000045
Frame Index: 185
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.004095
Depth frame count: 8
Color frame count: 7
Frame Index: 208
Color Captured.
Frame extracted.
Time since listen initiation: 0.000054
Frame Index: 212
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.004242
Depth frame count: 9
Color frame count: 8
Frame Index: 236
Color Captured.
Frame extracted.
Time since listen initiation: 0.000092
Frame Index: 240
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.005918
Depth frame count: 10
Color frame count: 9
Frame Index: 261
Color Captured.
Frame extracted.
Time since listen initiation: 0.000731
Frame Index: 262
Color Captured.
Frame extracted.
Time since listen initiation: 0.000877
Frame Index: 266
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.009347
Depth frame count: 11
Color frame count: 11
Frame Index: 286
Color Captured.
Frame extracted.
Time since listen initiation: 0.000047
Frame Index: 290
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.006080
Depth frame count: 12
Color frame count: 12
Frame Index: 311
Color Captured.
Frame extracted.
Time since listen initiation: 0.000072
Frame Index: 315
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.006453
Depth frame count: 13
Color frame count: 13
Frame Index: 337
Color Captured.
Frame extracted.
Time since listen initiation: 0.000062
Frame Index: 341
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007485
Depth frame count: 14
Color frame count: 14
Frame Index: 367
Color Captured.
Frame extracted.
Time since listen initiation: 0.000042
Frame Index: 371
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.003758
Depth frame count: 15
Color frame count: 15
Frame Index: 390
Color Captured.
Frame extracted.
Time since listen initiation: 0.000073
Frame Index: 395
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007917
Depth frame count: 16
Color frame count: 16
Frame Index: 416
Color Captured.
Frame extracted.
Time since listen initiation: 0.000105
Frame Index: 421
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007554
Depth frame count: 17
Color frame count: 17
Frame Index: 453
Color Captured.
Frame extracted.
Time since listen initiation: 0.000060
Frame Index: 458
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.006150
Depth frame count: 18
Color frame count: 18
Frame Index: 481
Color Captured.
Frame extracted.
Time since listen initiation: 0.000074
Frame Index: 486
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007169
Depth frame count: 19
Color frame count: 19
Frame Index: 517
Color Captured.
Frame extracted.
Time since listen initiation: 0.000045
Frame Index: 522
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.004196
Depth frame count: 20
Color frame count: 20
Frame Index: 547
Color Captured.
Frame extracted.
Time since listen initiation: 0.000071
Frame Index: 552
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007375
Depth frame count: 21
Color frame count: 21
Frame Index: 625
Color Captured.
Frame extracted.
Time since listen initiation: 0.000179
Frame Index: 631
Depth Captured.
Frame extracted.
Image overwritten.
Time since listen initiation: 0.007922
Depth frame count: 22
Color frame count: 22
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached
Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached
Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached
Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached
Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached
Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached
Unexpected stream: -1
Wait failed! (timeout is 2000 ms)
waitForStreams: timeout reached
Unexpected stream: -1
And here is the call to IMG_pipeline::listen(...):
IMG_pipeline pip_inst;
std::string FileName = "/home/derek/Test Data/RGBD/RGBD_S2_R1";
int Type = 2;
int Criteria = 1;
std::vector<cv::Mat> OUT;
int NumMissedIMGs;
int Start;
int Stop;
pip_inst.connect(FileName);
while (true)
{
pip_inst.listen(Type, Criteria);
if (pip_inst.IsNewIMG()){
OUT = pip_inst.GetImage();
cv::imshow("Current Frame", OUT.at(3));
char c = cv::waitKey(0);
if (c == 'f')
{
printf("Depth frame count: %i \n", pip_inst.GetDepthFrameCount());
printf("Color frame count: %i \n", pip_inst.GetColorFrameCount());
}
else
{
Start = pip_inst.GetColorFrameCount();
break;
}
cv::destroyWindow("Current Frame");
}
}
The color images are also alternating R, G, B tints. I am sure that is an issue with what order the data is in cv::Mat however.
Even more interesting, a call to IMG_pipeline::listen(...) that goes through many frames has a different index result then running IMG_pipeline::listen(...) multiple times, incrementing through the .oni file.
So the RGB information was not being copied correctly due to the variance of uchar from compiler to compiler. So the case statement for RGB information was changed to this:
case openni::PIXEL_FORMAT_RGB888:
{
openni::RGB888Pixel* imgbuffer = (openni::RGB888Pixel*)frame.getData();
//cvColor.data = (uchar*)imgbuffer;
memcpy( cvColor.data, imgbuffer, 3*frame.getHeight()*frame.getWidth()*sizeof(uint8_t));
cv::cvtColor(cvColor,cvColor,cv::COLOR_BGR2RGB);
TotalFrames[1]++;
ColorCaptured = true;
printf("Frame Index: %i \n", frame.getFrameIndex());
printf("Color Captured. \n");
break;
}
Additonally, to ensure that I am able to capture every frame, I slowed down the file play speed. Which is interesting that opening an .oni file plays at a speed rather than grabbing at frame indices (I can only assume this is the speed at which it was captured). Anyway, this is done with
Source.getPlaybackControl()->setSpeed(Ratio);
Where Source is my device and Ratio is a user specified float. Hope this helps someone in the future.