Where is pixel format stored in H.264 MP4 file?

I'm working on a transmuxer that will convert an H.264/AAC RTMP stream to a valid MP4 file. I'm mostly done. I'm parsing the AMF tag, reading the AVCDecoderConfigurationRecord and AACSpecificConfig, I'm generating a valid moov atom, etc.

After discovering and fixing a few bugs in my code, I've got a mostly valid MP4 file. However when I attempt to read the video in ffprobe I get the following error:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f9fb4000b80] Failed to open codec in avformat_find_stream_info
    Last message repeated 1 times
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f9fb4000b80] Could not find codec parameters for stream 1 (Video: h264 (avc1 / 0x31637661), none, 640x360): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options

It's unable to find the pixel format. Skimming through my AVCDecoderConfigurationRecord parsing logic (which is used to generate the avcC atom as part of the avc1 atom), I have the following:

// Parsed per: https://github.com/LiminWang/simple-rtmp-server/blob/master/trunk/doc/H.264-AVC-ISO_IEC_14496-15.pdf
var info = parseAVCConfig(packet);

// Fortunately my video sample has one of each of these
// I may need to concatenate multiple in the future
var sps = info.sps[0];
var pps = info.pps[0];

var avcc = box(
    types.avcC,
    new Uint8Array([
        // Version
        0x01,
        // Profile
        info.profile,
        // Profile Compat
        info.compat,
        // Level
        info.level,
        // LengthSizeMinusOne, hard-coded to 4 bytes (copied HLS.js)
        0xfc | 3,
        // 3bit reserved (111) + numOfSequenceParameterSets
        0xE0 | sps.byteLength
    ]
        .concat(Array.from(sps))
        .concat([
            // NumOfPictureParametersets
            pps.byteLength
        ])
        .concat(Array.from(pps))
    )
);

As you can see the avcc atom contains the profile, compat, and level -- but after that I just copy over the SPS and PPS directly from the AVCDecoderConfigurationRecord. Nowhere in the atom do I define a pixel format, so I assumed it was part of the SPS or PPS.

Looking at the spec for the AVCDecoderConfigurationRecord, there's nothing specifically called "pixel format", but there is a "chroma_format", "bit_depth_luma_minus8", and "bit_depth_chroma_minus_8" -- however these only exist if the profile is 100, 110, 122, or 244. My profile is 66 (and these bytes don't exist for me)

At the moment this proof of concept I'm doing only has to support a single video, so worst-case scenario I can hard-code the pixel format to yuv420. But I don't even know where to put this information in the output MP4. Does it go in the avcC atom? Or the avc1 atom? Or the mvhd atom?

Links:

buffer.mp4: This is the file I'm creating, which does not work. ffprobe says it cannot find the pixel format. http://files.stevendesu.com/buffer.mp4
test.mp4: This is a segment of the same video converted to MP4 by ffmpeg for comparison. http://files.stevendesu.com/test.mp4

Solution

Take a look at the chroma_format_idc Rec. ITU-T H.264 (04/2017) - 7.3.2.1.1 Sequence parameter set data syntax. chroma_format_idc is part of SPS. For profile_idc 100, 110, 122, 244, 44, 83, 86, 118, 128, 138, 139, 134 or 135 the chroma_format_idc is stored inside the SPS. Otherwise you assume 1 (= 4:2:0).

7.4.2.1.1 Sequence parameter set data semantics

chroma_format_idc specifies the chroma sampling relative to the luma sampling as specified in clause 6.2. The value of
chroma_format_idc shall be in the range of 0 to 3, inclusive. When chroma_format_idc is not present, it shall be inferred
to be equal to 1 (4:2:0 chroma format).