I'm trying to dynamically crop a video using FFmpeg's sendcmd filter based on coordinates specified in a text file, but the crop commands do not seem to be taking effect. Here's the format of the commands I've tried and the corresponding FFmpeg command I'm using.
Following the documentation https://ffmpeg.org/ffmpeg-filters.html#sendcmd_002c-asendcmd, commands in the text file (coordinates.txt) like this:
0.05 [enter] crop w=607:h=1080:x=0:y=0;
0.11 [enter] crop w=607:h=1080:x=0:y=0;
...
Ffmpeg command:
ffmpeg -i '10s.mp4' -filter_complex "[0:v]sendcmd=f=coordinates.txt" -c:v libx264 -c:a copy -r 30 output.mp4
This doesn’t seem to do anything.
And with the commands in the text file (coordinates.txt) like this:
0.05 crop w 607, crop h 1080, crop x 0, crop y 0;
0.11 crop w 607, crop h 1080, crop x 0, crop y 0;
...
Ffmpeg command:
ffmpeg -i '10s.mp4' -filter_complex "[0:v]sendcmd=f=coordinates.txt,crop" -c:v libx264 -c:a copy -r 30 output.mp4
(following this answer https://stackoverflow.com/a/67508233/1967110)
This one does something, but something very messy. It looks like it crops at the correct x, but does not take into account the y, w or h, and it puts the crop at the right side of the input video.
Edit: what I’m trying to do is create a 607x1080 (portrait format, 9:16) video from a 1920x1080 video, with the x parameter varying across time (imagine sliding horizontally a 9:16 frame over a 16:9 video). So fixed w, h and y, just x varying.
I’m using this ffmpeg version:
ffmpeg version 6.0 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
configuration: --prefix=/home/ffmpeg-builder/release --pkg-config-flags=--static --extra-libs=-lm --disable-doc --disable-debug --disable-shared --disable-ffprobe --enable-static --enable-gpl --enable-version3 --enable-runtime-cpudetect --enable-avfilter --enable-filters --enable-nvenc --enable-nvdec --enable-cuvid --toolchain=hardened --disable-stripping --enable-opengl --pkgconfigdir=/home/ffmpeg-builder/release/lib/pkgconfig --extra-cflags='-I/home/ffmpeg-builder/release/include -static-libstdc++ -static-libgcc ' --extra-ldflags='-L/home/ffmpeg-builder/release/lib -fstack-protector -static-libstdc++ -static-libgcc ' --extra-cxxflags=' -static-libstdc++ -static-libgcc ' --extra-libs='-ldl -lrt -lpthread' --enable-ffnvcodec --enable-gmp --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libkvazaar --enable-libmp3lame --enable-libopus --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libshine --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtheora --enable-libvidstab --ld=g++ --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-libzimg --enable-openssl --enable-zlib --enable-nonfree --extra-libs=-lpthread --enable-pthreads --extra-libs=-lgomp
Ok so the only command I got working so far, taken from https://video.stackexchange.com/posts/19403/revisions , is this:
ffmpeg -i in.mp4 -filter_complex_script file.txt -map "[out]" output.mp4
with file.txt like this
nullsrc=WxH:r=FPS[cv];
[cv][0]overlay=-X0:-Y0:shortest=1:enable='eq(n\,0)'[b0];
[b0][0]overlay=-X1:-Y1:shortest=1:enable='eq(n\,1)'[b1];
[b1][0]overlay=-X2:-Y2:shortest=1:enable='eq(n\,2)'[b2];
...
[bm-1][0]overlay=-Xm:-Ym:shortest=1:enable='eq(n\,m)'[out]
this takes about 40s to crop a 1920x1080 5s 30fps video to a 606x1080 video (on google colab, so low-level cpu, and without gpu). BUT it also causes Out of memory errors as soon as more than 900 frames need to be cropped, on a 12GB RAM machine (see https://superuser.com/questions/1839180/ffmpeg-out-of-memory-issues-with-filter-complex-script).
I also tried to first decompose the video into each of its frames, crop each frame, and reassemble the video. Here is the code:
def decompose_video(video_path, frames_dir):
if not os.path.exists(frames_dir):
os.makedirs(frames_dir)
command = [
'ffmpeg', '-i', video_path,
os.path.join(frames_dir, 'frame_%04d.png')
]
subprocess.run(command, check=True)
def read_crop_values(crop_file):
with open(crop_file, 'r') as file:
crops = [line.strip().split(',') for line in file]
return crops
def apply_crop(frames_dir, crops):
cropped_frames_dir = frames_dir + '_cropped'
if not os.path.exists(cropped_frames_dir):
os.makedirs(cropped_frames_dir)
for i, crop in enumerate(crops, 1):
x, y, w, h = crop
input_frame = os.path.join(frames_dir, f'frame_{i:04d}.png')
output_frame = os.path.join(cropped_frames_dir, f'frame_{i:04d}.png')
command = [
'ffmpeg', '-i', input_frame,
'-filter:v', f'crop={w}:{h}:{x}:{y}',
output_frame
]
subprocess.run(command, check=True)
def recreate_video(cropped_frames_dir, output_video_path):
command = [
'ffmpeg', '-y', '-loglevel', 'verbose', '-r', '29.97', '-f', 'image2', '-i',
os.path.join(cropped_frames_dir, 'frame_%04d.png'),
'-vcodec', 'libx264', '-crf', '23', '-pix_fmt', 'yuv420p',
output_video_path
]
try:
subprocess.run(command, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
except subprocess.CalledProcessError as e:
print("FFmpeg failed with error:")
print(e.stderr.decode())
video_path = 'input.mp4'
frames_dir = 'frames'
crop_file = 'crop_values.txt'
output_video_path = 'output.mp4'
decompose_video(video_path, frames_dir)
crops = read_crop_values(crop_file)
apply_crop(frames_dir, crops)
recreate_video(frames_dir + '_cropped', output_video_path)
and in crop_values.txt you just put x, y, w, h:
0,0,606,1080
8,0,606,1080
17,0,606,1080
It also works but is quite a bit slower (1m40 for the same video).