conversion from YUV to RGB color

Question

I'm trying to convert FFMPEG's Intensity color map from yuv format to rgb. It should give you colors as shown in the color bar in the image. You can generate a spectrogram using command:

ffmpeg -i words.wav -lavfi showspectrumpic=s=224x224:mode=separate:color=intensity spectrogram.png

Here is the code:

import numpy as np

def rgb_from_yuv(Y, U, V):

    Y -= 16
    U -= 128
    V -= 128
    R = 1.164 * Y + 1.596 * V
    G = 1.164 * Y - 0.392 * U - 0.813 * V
    B = 1.164 * Y + 2.017 * U
  
    # Clip and normalize RGB values
    R = np.clip(R, 0, 255)
    G = np.clip(G, 0, 255)
    B = np.clip(B, 0, 255)
  
    return '#{:02X}{:02X}{:02X}'.format(int(R), int(G), int(B))

def yuv_to_rgb(Y, U,V):
    # Convert YUV to RGB
    R  = Y + (V - 128) *  1.40200
    G  = Y + (U - 128) * -0.34414 + (V - 128) * -0.71414
    B  = Y + (U - 128) *  1.77200

    # Clip and normalize RGB values
    R = np.clip(R, 0, 255)
    G = np.clip(G, 0, 255)
    B = np.clip(B, 0, 255)
    
    print(R,G,B)
    
    return '#{:02X}{:02X}{:02X}'.format(int(R), int(G), int(B))

# FFMPEG's intensity color map
colors = [
    [    0,                  0,                  0,                   0 ],
    [ 0.13, .03587126228984074,  .1573300977624594, -.02548747583751842 ],
    [ 0.30, .18572281794568020,  .1772436246393981,  .17475554840414750 ],
    [ 0.60, .28184980583656130, -.1593064119945782,  .47132074554608920 ],
    [ 0.73, .65830621175547810, -.3716070802232764,  .24352759331252930 ],
    [ 0.78, .76318535758242900, -.4307467689263783,  .16866496622310430 ],
    [ 0.91, .95336363636363640, -.2045454545454546,  .03313636363636363 ],
    [    1,                  1,                  0,                   0 ]]
    
cmaps = []
for i, c in enumerate(colors):
    Y = c[1]
    U = c[2]
    V = c[3]

    hex = yuv_to_rgb(Y,U,V)

    cmaps.append((c[0], hex))    
print(cmaps)

Both the functions are not providing desired output.

Answer 1

It appears that the U and V channels pixel values are accumulated from 127. See the FFmpeg source:

https://github.com/FFmpeg/FFmpeg/blob/7e59c4f90885a9ffffb0a3f1d385b4eae3530529/libavfilter/avf_showspectrum.c#L1416

So, I suspect if you add 0.5 to the 3rd and 4th columns of colors first, the standard YUV-to-RGB conversion should work.

edit 4/13/24: Got it.

rgb = [[  0   0   0]
       [  0   0  71]
       [106   0 125]
       [255   0   0]
       [255 165   0]
       [255 218   0]
       [255 255 159]
       [255 255 255]]

I reverse-engineered how showspectrumpic picks a YUV color value on its legend and passed these YUV values to FFmpeg in yuv444p format and converted the frame as rgb24.

import ffmpegio as ff # caveat: need the GitHub version as of 4/13/24
import numpy as np
from matplotlib import pyplot as plt

color_table = np.array(
    [  # a y u v
        [0, 0, 0, 0],
        [0.13, 0.03587126228984074, 0.1573300977624594, -0.02548747583751842],
        [0.30, 0.18572281794568020, 0.1772436246393981, 0.17475554840414750],
        [0.60, 0.28184980583656130, -0.1593064119945782, 0.47132074554608920],
        [0.73, 0.65830621175547810, -0.3716070802232764, 0.24352759331252930],
        [0.78, 0.76318535758242900, -0.4307467689263783, 0.16866496622310430],
        [0.91, 0.95336363636363640, -0.2045454545454546, 0.03313636363636363],
        [1, 1, 0, 0],
    ]
)

# U & V channels are offset values from the middle
cmap_yuv = np.clip(color_table[:, 1:] + np.array([0, 0.5, 0.5]), 0, 1)

# convert to YUV444P planar pixel format
cmap_yuvp = (cmap_yuv*255).astype("uint8").transpose()

# convert to RGB24
cmap_rgb = ff.image.filter(
    None, cmap_yuvp.reshape([3, 1, -1]),
    pix_fmt_in="yuv444p",
    s_in=(1, len(cmap_yuv)),
)
# equiv to: ffmpeg -f rawvideo -c:v rawvideo -pix_fmt yuv444p -r 1 -s 1x8 -i - -f rawvideo -pix_fmt rgb24 -

print(cmap_rgb)

plt.imshow(cmap_rgb.reshape(1, -1, 3))
plt.show()

I've used my ffmpegio package to shortcut the FFmpeg call (note: I had to make a slight change to the code so only the GitHub version works at this time).

Finally, be aware this colormap is not equispaced. The map intensity needs to be normalized to (0,1) then be linear-interpolated according to the "A" column.