Access crop bounds for decoded video using NDK MediaCodec

Question

I am trying to read crop bounds via NDK's MediaCodec API.

I am reading values as below:

AMediaFormat* fmt = initialized();
if (AMediaFormat_getInt32(fmt, "crop-left", &cropLeft)) {
    LOGI("crop-left: %d", cropLeft);
}

However, I fail to read cropLeft values for various videos that I tried. I have tried to read it after first frame and also once format change is detected.

I have seen similar code being used in vlc code base.

Looking for what might be potentially wrong here.

Answer 1

It's not possible yet to read the crop values. Under the hood the AMediaFormat contains a message struct (AMessage) that is send by stagefright with all the crop values. The crop value is of special type Rect, that needs a new read method besides the ones already defined in AMediaFormat (int32, int64, string, buffer). Unfortunately, the Rect type is even not handled by the AMediaFormat_toString method. We can see an output as "crop: unknown(9)" instread of the crop values.

UPDATE:

Here is an ugly hack I am using on Galaxy S6 with Android 7.0. It can't be used in production code. It's just for testing and demos. It will crash on other Android versions and Vendor phones. Structs are expected to change in other Android versions and so the offsets, paddings and other values needs to be fixed. Anyway, the AMediaFormat is a struct which has a pointer to AMessage. It is expected that this is in the first 4 bytes. AMessage is a subclass of RefBase and with mainoffset[24] we are stepping some bytes forward to the Items array in AMessage class. We are expecting the Items struct has a size of 32 bytes and it has a char pointer to the "crop" string. When we found the crop item then we can read out the crop values.

AMediaFormat* format = AMediaCodec_getOutputFormat(_codec);

typedef struct {
 char mainoffset[24];
 struct {
  char itemoffset[8];
  int32_t cropLeft;
  int32_t cropTop;
  int32_t cropRight;
  int32_t cropBottom;
  unsigned char* name;
  char padding[4];
 }items[64];
}AMessage;

AMessage* amessage = *((AMessage**)format);

int n = 0;
while ( amessage->items[n].name ) {
 if ( !strcmp((const char*)amessage->items[n].name, "crop") ) {
  int32_t width = amessage->items[n].cropRight - amessage->items[n].cropLeft + 1;
  int32_t height = amessage->items[n].cropBottom - amessage->items[n].cropTop + 1;
  __android_log_print(ANDROID_LOG_VERBOSE, "FORMAT", "width=%d height=%d", width, height);
  break;
 }    
 n++;
}