Search code examples
dockercurlrustffmpegrust-tokio

ffmpeg Command in Docker with Rust Tokio Closes Warp Server Connection (curl 52 Error)

I’m encountering an issue where executing an ffmpeg concatenation command through Rust’s Tokio process in a Docker container causes subsequent HTTP requests to fail. The error occurs exclusively after running the ffmpeg command and making immediate requests, resulting in a “curl 52 empty response from server” error with the connection being closed. Notably, this issue does not occur when running the same setup outside of Docker. Additionally, if no HTTP requests are made after the ffmpeg command, the curl 52 error does not occur.

Here is the verbose curl output of my minimum reproducible example (see below).

curl -v "http://localhost:3030"
*   Trying 127.0.0.1:3030...
* Connected to localhost (127.0.0.1) port 3030 (#0)
> GET / HTTP/1.1
> Host: localhost:3030
> User-Agent: curl/8.1.2
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

Here are Docker logs from my minimum reproducible example (see below). The wav files are concatenated successfully, then the container appears to rebuild.

[2024-06-03T05:26:58Z INFO  minimal_docker_webserver_post_error] Starting server on 0.0.0.0:3030
[2024-06-03T05:26:58Z INFO  warp::server] Server::run; addr=0.0.0.0:3030
[2024-06-03T05:26:58Z INFO  warp::server] listening on http://0.0.0.0:3030
[2024-06-03T05:27:07Z INFO  minimal_docker_webserver_post_error] WAV files concatenated successfully
[Running 'cargo run']
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.06s
     Running `target/debug/minimal_docker_webserver_post_error`
[2024-06-03T05:27:08Z INFO  minimal_docker_webserver_post_error] Starting server on 0.0.0.0:3030
[2024-06-03T05:27:08Z INFO  warp::server] Server::run; addr=0.0.0.0:3030
[2024-06-03T05:27:08Z INFO  warp::server] listening on http://0.0.0.0:3030

What I have tried: I tried using different web frameworks (Warp, Actix-web) and request crates (reqwest, ureq). I also tried running the setup outside of Docker, which worked as expected without any issues. Additionally, I tried running the setup in Docker without making any HTTP requests after the ffmpeg command, and the connection closed successfully without errors. I also tried posting to httpbin with a minimal request, but the issue persisted.

Minimum reproducible example:

main.rs

use warp::Filter;
use reqwest::Client;
use std::convert::Infallible;
use log::{info, error};
use env_logger;
use tokio::process::Command;

#[tokio::main]
async fn main() {
    std::env::set_var("RUST_LOG", "debug");
    env_logger::init();

    let route = warp::path::end()
        .and_then(handle_request);

    info!("Starting server on 0.0.0.0:3030");
    warp::serve(route)
        .run(([0, 0, 0, 0], 3030))
        .await;
}

async fn handle_request() -> Result<impl warp::Reply, Infallible> {
    let client = Client::new();

    let output = Command::new("ffmpeg")
        .args(&[
            "y",
            "-i", "concat:/usr/src/minimal_docker_webserver_post_error/file1.wav|/usr/src/minimal_docker_webserver_post_error/file2.wav",
            "-c", "copy",
            "/usr/src/minimal_docker_webserver_post_error/combined.wav"
        ])
        .output()
        .await;

    match output {
        Ok(output) => {
            if output.status.success() {
                info!("WAV files concatenated successfully");
            } else {
                error!("Failed to concatenate WAV files: {:?}", output);
                return Ok(warp::reply::with_status("Failed to concatenate WAV files", warp::http::StatusCode::INTERNAL_SERVER_ERROR));
            }
        },
        Err(e) => {
            error!("Failed to execute ffmpeg: {:?}", e);
            return Ok(warp::reply::with_status("Failed to execute ffmpeg", warp::http::StatusCode::INTERNAL_SERVER_ERROR));
        }
    }

    // ISSUE: Connection closes with curl: (52) Empty reply from server
    match client.get("https://httpbin.org/get").send().await {
        Ok(response) => info!("GET request successful: {:?}", response),
        Err(e) => error!("GET request failed: {:?}", e),
    }

    match client.post("https://httpbin.org/post")
        .body("field1=value1&field2=value2")
        .send().await {
        Ok(response) => info!("POST request successful: {:?}", response),
        Err(e) => error!("POST request failed: {:?}", e),
    }

    Ok(warp::reply::with_status("Request handled", warp::http::StatusCode::OK))
}

FFMPEG command to generate the two wav files for concatenation

ffmpeg -f lavfi -i "sine=frequency=1000:duration=5" file1.wav && ffmpeg -f lavfi -i "sine=frequency=500:duration=5" file2.wav

Dockerfile

# Use the official Rust image as the base image
FROM rust:latest

# Install cargo-watch
RUN cargo install cargo-watch

# Install ffmpeg
RUN apt-get update && apt-get install -y ffmpeg

# Set the working directory inside the container
WORKDIR /usr/src/minimal_docker_webserver_post_error

# Copy the Cargo.toml and Cargo.lock files
COPY Cargo.toml Cargo.lock ./

# Copy the source code
COPY src ./src

# Copy wav files
COPY file1.wav /usr/src/minimal_docker_webserver_post_error/file1.wav
COPY file2.wav /usr/src/minimal_docker_webserver_post_error/file2.wav

# Install dependencies
RUN cargo build --release

# Expose the port that the application will run on
EXPOSE 3030

# Set the entry point to use cargo-watch
CMD ["cargo", "watch", "-x", "run"]

Cargo.toml

[package]
name = "minimal_docker_webserver_post_error"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
warp = "0.3"
reqwest = { version = "0.12.4", features = ["json"] }
tokio = { version = "1", features = ["full"] }
log = "0.4"
env_logger = "0.11.3"

Making the request to the warp server

curl -v "http://localhost:3030"
Solution

  • This behavior is caused by using cargo watch and your FFmpeg command re-triggering it.

    Your Dockerfile launches the server with cargo watch -x run, which is poised to rerun cargo run whenever the local directory is modified - which is /usr/src/minimal_docker_webserver_post_error. The command being ran on your Warp endpoint will create/modify combined.wav in that directory. This will cause the current server to be killed by cargo watch and restarted. This is what you see in your server logs.

    The reason this seems to work in isolation but causes a curl (52) error when an HTTP call is introduced is just due to timing. When there is no HTTP call, your Warp endpoint is able to write the response immediately and thus the response is seen fully by curl before cargo watch kills the server. But when there is an HTTP call, there is a significant delay (at least many milliseconds) such that the cargo watch kills the server before it resolves. This means the Warp endpoint did not yet construct the response to curl and the connection was severed, which is reported by curl as "Empty reply from server".

    All-in-all cargo watch is a fine tool, but I would not recommend using it in this scenario. A cargo run should suffice; you can and should configure Docker itself to auto-restart the container if that is desired.