I want to display cropped and scaled frames using the ffmpeg api and I am using GTK+ 3 for the GUI components. From following the this tutorial and the ffmpeg examples, I am able to display unfiltered frames, though with some instability. The filtered frames does not display correctly at all. It mostly produces completely black output. I suspect that this is due to sws_scale() but I have not found out why this is happening.
Using the "trivial" display from the ffmpeg examples I can confirm that the frame is being cropped and scaled properly.
Running the code below I get a bunch of errors:
[swscaler @ 0x7fb58b025400] bad src image pointers
[swscaler @ 0x7fb58b025400] bad dst image pointers
I also get this error:
[swscaler @ 0x7fd05c025600] Warning: data is not aligned! This can lead to a speedloss
I tried making a buffer that was 16 bit aligned, but it did not seem to have any effect on the result.
This is how I decode the frames and apply the filters:
void decode(gpointer args) {
int ret;
AVPacket packet;
AVFrame *frame = av_frame_alloc();
AVFrame *filt_frame = av_frame_alloc();
int got_frame;
if(!frame || !filt_frame) {
perror("Could not allocate frame");
exit(1);
}
/* read all packets */
while (1) {
if ((ret = av_read_frame(fmt_ctx, &packet)) < 0)
break;
if (packet.stream_index == video_stream_index) {
got_frame = 0;
ret = avcodec_decode_video2(dec_ctx, frame, &got_frame, &packet);
if (ret < 0) {
av_log(NULL, AV_LOG_ERROR, "Error decoding video\n");
break;
}
if (got_frame) {
frame->pts = av_frame_get_best_effort_timestamp(frame);
/* push the decoded frame into the filtergraph */
if (av_buffersrc_add_frame_flags(buffersrc_ctx, frame, AV_BUFFERSRC_FLAG_KEEP_REF) < 0) {
av_log(NULL, AV_LOG_ERROR, "Error while feeding the filtergraph\n");
break;
}
/* pull filtered frames from the filtergraph */
while (1) {
ret = av_buffersink_get_frame(buffersink_ctx, filt_frame);
if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
break;
if (ret < 0)
goto end;
display_frame2(filt_frame, buffersink_ctx->inputs[0]->time_base);
av_frame_unref(filt_frame);
}
av_frame_unref(frame);
}
}
av_free_packet(&packet);
}
end:
avfilter_graph_free(&filter_graph);
avcodec_close(dec_ctx);
avformat_close_input(&fmt_ctx);
av_frame_free(&frame);
av_frame_free(&filt_frame);
if (ret < 0 && ret != AVERROR_EOF) {
fprintf(stderr, "Error occurred: %s\n", av_err2str(ret));
exit(1);
}
}
And this is how I display the frames.
void display_frame2(const AVFrame *frame, AVRational time_base) {
GdkPixbuf *pixbuf;
int64_t delay;
AVFrame *filt;
uint8_t *buffer;
int num_bytes, i;
buffer = NULL;
filt = av_frame_alloc();
num_bytes = avpicture_get_size(PIX_FMT_RGB24, dec_ctx->width, dec_ctx->height);
buffer = (uint8_t *)av_malloc(num_bytes * sizeof(uint8_t));
avpicture_fill((AVPicture *)filt, buffer, PIX_FMT_RGB24, dec_ctx->width, dec_ctx->height);
if (frame->pts != AV_NOPTS_VALUE) {
if (last_pts != AV_NOPTS_VALUE) {
/* sleep roughly the right amount of time;
* usleep is in microseconds, just like AV_TIME_BASE. */
delay = av_rescale_q(frame->pts - last_pts,
time_base, AV_TIME_BASE_Q);
if (delay > 0 && delay < 1000000)
usleep(delay);
}
last_pts = frame->pts;
}
sws_scale( sws_ctx,
(uint8_t const * const *)frame->data,
frame->linesize,
0,
frame->height,
filt->data,
filt->linesize);
pixbuf = gdk_pixbuf_new_from_data( filt->data[0], GDK_COLORSPACE_RGB,
0, 8, dec_ctx->width, dec_ctx->height,
filt->linesize[0], NULL, NULL);
gtk_image_set_from_pixbuf((GtkImage *)image, pixbuf);
free( filt );
free( buffer );
}
EDIT: After some more thought and experimentation I got the filtered frames to be displayed, albeit in SDL, not GTK+. I used the transcoding example from ffmpeg to see if I could re-encode the video with the filters, and that does indeed work. With that example I basically changed the filter being fed the filtergtaph and most of the work is already done. At this point all I am doing is to display the video using SDL as shown in danger's tutorial. The cropping filter creates a lot of artifacts but it is at least showing something.
I have to do some more work to see if it will work with GTK+. I have not taken a detailed look at the differences between the above program and the one in the transcoding example, so I have not yet figured out why my old code does not display filtered frames. Both sets of code use sws_scale() but I get no errors with the new code, so that must mean that something is different. I will update this post once I make more progress.
EDIT 2: Added a small compilable example that should work, as per @drahnr’s request. I have not had the chance to try out replacing GtkPixbuf.
#define _XOPEN_SOURCE 600
#include <libavformat/avformat.h>
#include <libavformat/avio.h>
#include <libavfilter/avfiltergraph.h>
#include <libavfilter/avcodec.h>
#include <libavfilter/buffersink.h>
#include <libavfilter/buffersrc.h>
#include <libavcodec/avcodec.h>
#include <libswscale/swscale.h>
#include <libavutil/avstring.h>
#include <libavutil/time.h>
#include <libavutil/opt.h>
#include <unistd.h>
#include <gtk/gtk.h>
#include <gdk/gdkx.h>
GtkWidget *image;
GtkWidget *window;
struct SwsContext *sws_ctx;
char *filter_descr = "crop=100:100,scale=640:360";
static AVFormatContext *fmt_ctx;
static AVCodecContext *dec_ctx;
AVFilterContext *buffersink_ctx;
AVFilterContext *buffersrc_ctx;
AVFilterGraph *filter_graph;
static int video_stream_index = -1;
static void open_input_file(const char *filename)
{
AVCodec *dec;
avformat_open_input(&fmt_ctx, filename, NULL, NULL);
avformat_find_stream_info(fmt_ctx, NULL);
video_stream_index = av_find_best_stream(fmt_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, &dec, 0);
dec_ctx = fmt_ctx->streams[video_stream_index]->codec;
av_opt_set_int(dec_ctx, "refcounted_frames", 1, 0);
avcodec_open2(dec_ctx, dec, NULL);
}
static void init_filters(const char *filters_descr)
{
char args[512];
AVFilter *buffersrc = avfilter_get_by_name("buffer");
AVFilter *buffersink = avfilter_get_by_name("buffersink");
AVFilterInOut *outputs = avfilter_inout_alloc();
AVFilterInOut *inputs = avfilter_inout_alloc();
AVRational time_base = fmt_ctx->streams[video_stream_index]->time_base;
enum AVPixelFormat pix_fmts[] = { PIX_FMT_RGB24, AV_PIX_FMT_NONE };
filter_graph = avfilter_graph_alloc();
snprintf(args, sizeof(args),
"video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",
dec_ctx->width, dec_ctx->height, dec_ctx->pix_fmt,
time_base.num, time_base.den,
dec_ctx->sample_aspect_ratio.num, dec_ctx->sample_aspect_ratio.den);
avfilter_graph_create_filter(&buffersrc_ctx, buffersrc, "in", args, NULL, filter_graph);
avfilter_graph_create_filter(&buffersink_ctx, buffersink, "out", NULL, NULL, filter_graph);
av_opt_set_int_list(buffersink_ctx, "pix_fmts", pix_fmts, AV_PIX_FMT_NONE, AV_OPT_SEARCH_CHILDREN);
outputs->name = av_strdup("in");
outputs->filter_ctx = buffersrc_ctx;
outputs->pad_idx = 0;
outputs->next = NULL;
inputs->name = av_strdup("out");
inputs->filter_ctx = buffersink_ctx;
inputs->pad_idx = 0;
inputs->next = NULL;
avfilter_graph_parse_ptr(filter_graph, filters_descr, &inputs, &outputs, NULL);
avfilter_graph_config(filter_graph, NULL);
}
static void display_frame2(const AVFrame *frame, AVRational time_base) {
GdkPixbuf *pixbuf;
AVFrame *filt;
uint8_t *buffer;
int num_bytes;
buffer = NULL;
filt = av_frame_alloc();
num_bytes = avpicture_get_size(PIX_FMT_RGB24, dec_ctx->width, dec_ctx->height);
buffer = (uint8_t *)av_malloc(num_bytes * sizeof(uint8_t));
avpicture_fill((AVPicture *)filt, buffer, PIX_FMT_RGB24, dec_ctx->width, dec_ctx->height);
usleep(33670 / 4);
sws_scale( sws_ctx,
(uint8_t const * const *)frame->data,
frame->linesize,
0,
frame->height,
filt->data,
filt->linesize);
pixbuf = gdk_pixbuf_new_from_data( filt->data[0], GDK_COLORSPACE_RGB,
0, 8, dec_ctx->width, dec_ctx->height,
filt->linesize[0], NULL, NULL);
gtk_image_set_from_pixbuf((GtkImage *)image, pixbuf);
free( filt );
free( buffer );
}
void decode(gpointer args) {
int ret;
AVPacket packet;
AVFrame *frame = av_frame_alloc();
AVFrame *filt_frame = av_frame_alloc();
int got_frame;
while (1) {
av_read_frame(fmt_ctx, &packet);
if (packet.stream_index == video_stream_index) {
got_frame = 0;
avcodec_decode_video2(dec_ctx, frame, &got_frame, &packet);
if (got_frame) {
frame->pts = av_frame_get_best_effort_timestamp(frame);
if (av_buffersrc_add_frame_flags(buffersrc_ctx, frame, AV_BUFFERSRC_FLAG_KEEP_REF) < 0) {
av_log(NULL, AV_LOG_ERROR, "Error while feeding the filtergraph\n");
break;
}
while (1) {
ret = av_buffersink_get_frame(buffersink_ctx, filt_frame);
if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
break;
// Display original frame
display_frame2(frame, buffersink_ctx->inputs[0]->time_base);
// Display filtered frame
// display_frame2(filt_frame, buffersink_ctx->inputs[0]->time_base);
av_frame_unref(filt_frame);
}
av_frame_unref(frame);
}
}
av_free_packet(&packet);
}
}
static void realize_cb(GtkWidget *widget, gpointer data) {
GThread *tid;
tid = g_thread_new("video", decode, NULL);
}
static void destroy(GtkWidget *widget, gpointer data) {
gtk_main_quit();
}
int main(int argc, char **argv)
{
av_register_all();
avfilter_register_all();
open_input_file(argv[1]);
init_filters(filter_descr);
sws_ctx = NULL;
sws_ctx = sws_getContext( dec_ctx->width, dec_ctx->height, dec_ctx->pix_fmt, dec_ctx->width, dec_ctx->height,
PIX_FMT_RGB24, SWS_BILINEAR, NULL, NULL, NULL );
av_dump_format( fmt_ctx, 0, argv[1], 0);
gtk_init(&argc, &argv);
window = gtk_window_new( GTK_WINDOW_TOPLEVEL );
g_signal_connect(window, "realize", G_CALLBACK(realize_cb), NULL);
g_signal_connect(window, "destroy", G_CALLBACK(destroy), NULL);
gtk_container_set_border_width(GTK_CONTAINER(window), 10);
image = gtk_image_new();
gtk_widget_show(image);
gtk_container_add(GTK_CONTAINER(window), image);
gtk_widget_show(window);
gtk_main();
return 0;
}
The proper way to do this is to use a cairo_surface_t and a bare metal GtkDrawingArea. Get its size allocation, scale the content accordingly (here: your video frame).
So when do you want to paint? Surely not as you do by now.
Sync it to the frame clock, and read the according doc pages (yes this is work but it will help you quite a bit) https://developer.gnome.org/gdk3/stable/GdkFrameClock.html#gdk-frame-clock-begin-updating
and schedule redraws whenever a new frame is ready to be displayed.
The major error I see with your above program is that you access a UI element GtkImage *image from a thread that is not the thread running the gtk_main/g_event_loop which is not allowed since only a very small subset of g/gtk_ calls are threads safe. One of those to circumvent is g_idle_add which enqueues a callback with data into the mainloop (search for GSource in the gtk docs).
For the alignment, I have no clue what's the issue here, I also got the warning even when aligning it with posix_memalign to 512 bytes
EDIT: as it seems, you the stride also has to be a multiple of the 16 bytes which is the required alignment.
Started a clean draft widget implementation at github, though this does not compile yet - but still it outlines the approach pretty clearly.