some codecs (ie. libx264) cannot be reused after draining

Question

After draining an encoder (by sending it a null frame and then receiving packets until EOF) it enters draining mode after which avcodec_send_frame will fail, returning EOF. You're supposed to call avcodec_flush_buffers which among other things sets the internal draining flag to 0, allowing avcodec_send_frame to work again. Unfortunately it checks AV_CODEC_CAP_ENCODER_FLUSH first and if not set it returns without resetting the draining flag.

So the only way to reuse the codec is to close and reopen it, which is wasteful, plus avcodec_close is deprecated. Am I missing something?

int attribute_align_arg avcodec_send_frame(AVCodecContext *avctx, const AVFrame *frame)
{
    AVCodecInternal *avci = avctx->internal;
    int ret;

    if (!avcodec_is_open(avctx) || !av_codec_is_encoder(avctx->codec))
        return AVERROR(EINVAL);

    if (avci->draining)
        return AVERROR_EOF;

    if (avci->buffer_frame->buf[0])
        return AVERROR(EAGAIN);

    if (!frame) {
        avci->draining = 1;
    } else {
        ret = encode_send_frame_internal(avctx, frame);
        if (ret < 0)
            return ret;
    }

    if (!avci->buffer_pkt->data && !avci->buffer_pkt->side_data) {
        ret = encode_receive_packet_internal(avctx, avci->buffer_pkt);
        if (ret < 0 && ret != AVERROR(EAGAIN) && ret != AVERROR_EOF)
            return ret;
    }

    avctx->frame_num++;

    return 0;
}

void avcodec_flush_buffers(AVCodecContext *avctx)
{
    AVCodecInternal *avci = avctx->internal;

    if (av_codec_is_encoder(avctx->codec)) {
        int caps = avctx->codec->capabilities;

        if (!(caps & AV_CODEC_CAP_ENCODER_FLUSH)) {
            // Only encoders that explicitly declare support for it can be
            // flushed. Otherwise, this is a no-op.
            av_log(avctx, AV_LOG_WARNING, "Ignoring attempt to flush encoder "
                   "that doesn't support it\n");
            return;
        }
        ff_encode_flush_buffers(avctx);
    } else
        ff_decode_flush_buffers(avctx);

    avci->draining      = 0;
    avci->draining_done = 0;
    if (avci->buffer_frame)
        av_frame_unref(avci->buffer_frame);
    if (avci->buffer_pkt)
        av_packet_unref(avci->buffer_pkt);

    if (HAVE_THREADS && avctx->active_thread_type & FF_THREAD_FRAME &&
        !avci->is_frame_mt)
        ff_thread_flush(avctx);
    else if (ffcodec(avctx->codec)->flush)
        ffcodec(avctx->codec)->flush(avctx);
}

Answer 1

The doxy for avcodec_flush_buffers says,

this function will only do something if the encoder declares support for AV_CODEC_CAP_ENCODER_FLUSH. When called, the encoder will drain any remaining packets, and can then be re-used for a different stream (as opposed to sending a null frame which will leave the encoder in a permanent EOF state after draining).

So,

1. the encoder has to declare support for AV_CODEC_CAP_ENCODER_FLUSH, lavc/libx264 does not. Only some GPU encoders do.
2. Even if it did, sending a NULL frame will trigger permanent EOF.