FFmpeg av_hwdevice_ctx_create returns ENOMEM

Question

I am trying to perform h264 video encoding on a GPU using the ffmpeg's vaapi library.

I have been following this example from ffmpeg Github repo.

It fails for me here:

AVBufferRef* vaapi_encoder;
int err = av_hwdevice_ctx_create(&vaapi_encoder, AV_HWDEVICE_TYPE_VAAPI,
                                 NULL, NULL, 0);

This returns -12, which maps to ENOMEM (out of memory) error, but that doesn't make sense to me because I have plenty of memory.

For reference, this is my output after running 'vainfo':

libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 20.1.1 ()
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD

Thanks!

Answer 1

Ok, so I figured out what was the issue.

Even though I had the right FFmpeg configurations installed on my system (the one with --enable-hwaccel=h264_vaapi), the issue was that I was linking my program with the wrong version of libav, the one contained in libwebrtc, not the one from my system.

So my incorrect Makefile had something like:

LDLIBS += -lwebrtc -lpthread -lm -lavcodec -lavutil -lavformat -lavdevice

So it was linking with libwebrtc first, and using the libav stuff from there.

After I moved -lwebrtc to link after -lavcodec -lavutil, it started working:

LDLIBS += -lpthread -lm -lavcodec -lavutil -lavformat -lavdevice -lwebrtc

Also, I made sure to run the program with sudo, or else it would give me other errors.