Search code examples
videoffmpegvideo-processingyuvcolor-space

Why are Cb and Cr planes displaced differently from lum by the displace complex filter in ffmpeg?

I have a video encoded with the yuv420p pixel format and I want to displace its pixels. I'm using ffmpeg and its new displace filter. The filter takes as inputs (the video to be displaced and) two displacement maps respectively for X and Y axis. I decided to create the displacement maps directly into ffmpeg using the nullsrc video source filter and the geq filter to specify the value of the three planes: lum, Cb, Cr. The script is the following:

ffmpeg INPUT.mp4 -f lavfi -i nullsrc=size=${WIDTH}x${HEIGHT}:d=0.1,geq='lum=128+30*sin(2*PI*X/400):Cb=128+30*sin(2*PI*X/400):Cr='128+30*sin(2*PI*X/400)' -f lavfi -i nullsrc=size=${WIDTH}x${HEIGHT}:d=0.1,geq='lum=128+30*sin(2*PI*X/400):Cb=128+30*sin(2*PI*X/400):Cr=128+30*sin(2*PI*X/400)' -lavfi '[0][1][2]displace' OUTPUT.mp4

I used the example provided in the documentation of ffmpeg, since the expression used in geq is irrelevant for the purposes of the problem.

At the and of the computation, I get the pixels of the input video not properly displaced, meaning that I can clearly see a sort of ghost carrying-color-information video under a displaced but b/w one. After some tests, I noticed that the displacemnt map created had only the luma plane displaced correctly while the chrominance planes were displaced, but differently from luma, which is the origin of the planes disalignment in the intput video as you can see in the following extract frames:

enter image description here

I also noticed that the video describing the Cb and Cr planes of the displacement maps have half resolution of the luma plane.

My question is: how can i setup correctly the Cr and Cb planes in the geq definition so that they are exactly identical to the luma plane?

It would be also great if someone could explain me why ffmpeg gives me an output so much different for luma and Cb, Cr planes even if the function provided is the same.

If, it can help, i'm using ffmpeg 3.3-static build.

Thanks for your time.

Solution

  • YUV420P means that Cb and Cr are sampled, as you discovered, at half the rate. Each Cb/Cr pixel/sample is applied to 4 pixels (2x2, in this case). Your expression displaces each chroma sample by the same amount as the luma, which has the effect of doubling the effective displacement. Workaround is to rescale the input and the displacement maps to fully sampled chroma planes and downsample, if desired, to 4:2:0 at the end of the filtering.

    ffmpeg -i in.mp4 -f lavfi -i nullsrc=size=1280x720:d=0.1,format=yuv444p,geq=lum='128+30*sin(2*PI*X/400)':cb='(128+30*sin(2*PI*X/400))':cr='(128+30*sin(2*PI*X/400))' -lavfi [0]format=yuv444p[v];[v][1][1]displace,format=yuv420p out.mp4

    For working on YUV420P input, without any pixel format conversion, use

    ffmpeg -i in.mp4 -f lavfi -i nullsrc=size=1280x720:d=0.1,geq=lum='128+30*sin(2*PI*X/400)':cb='(128+15*sin(2*PI*2*X/400))':cr='(128+15*sin(2*PI*2*X/400))' -lavfi [0][1][1]displace out.mp4