I am trying to capture the frame buffer for an arbitrary application on Mac and process the frame buffers, frame by frame. Is there a way to tap into an applications internal frame buffer? I also noticed there are some Graphics API functions like this that can capture the entire screen, but I worry that
A) I'm going to need to figure out where my window is and crop it out of the entire screen image which may take a non-zero amount of time
B) Since it's capturing the entire screen instead of just one window, I imagine it'll take a longer amount of time.
I'm hoping to be able to capture and process 20 frames per second, so speed is pretty important.
I have a couple of ideas...
Firstly, and most simply, you can start Quicktime (which is supplied with macOS) and go to File->New screen recording and record an arbitrary area of the screen and save it in a movie and analyse the frames later.
Secondly, you can use screencapture (/usr/sbin/screencapture) and either specify a rectangle to capture or a window id. The manual page is incorrect and rubbish, so use the following to see the actual options:
screencapture -h
Output
usage: screencapture [-icMPmwsWxSCUtoa] [files]
-c force screen capture to go to the clipboard
-b capture Touch Bar - non-interactive modes only
-C capture the cursor as well as the screen. only in non-interactive modes
-d display errors to the user graphically
-i capture screen interactively, by selection or window
control key - causes screen shot to go to clipboard
space key - toggle between mouse selection and
window selection modes
escape key - cancels interactive screen shot
-m only capture the main monitor, undefined if -i is set
-M screen capture output will go to a new Mail message
-o in window capture mode, do not capture the shadow of the window
-P screen capture output will open in Preview
-I screen capture output will in a new Messages message
-s only allow mouse selection mode
-S in window capture mode, capture the screen not the window
-t<format> image format to create, default is png (other options include pdf, jpg, tiff and other formats)
-T<seconds> Take the picture after a delay of <seconds>, default is 5
-w only allow window selection mode
-W start interaction in window selection mode
-x do not play sounds
-a do not include windows attached to selected windows
-r do not add dpi meta data to image
-l<windowid> capture this windowsid
-R<x,y,w,h> capture screen rect
-B<bundleid> screen capture output will open in app with bundleidBS
files where to save the screen capture, 1 file per screen
As you can see, the -l, -R options are very useful.
I wrote a little program to get a list of windowids in another answer, here.
The size of the window and the file format make a difference to the speed. I find JPEG is normally fastest, and PNG is normally slowest. I can get 20 frames a second with a reasonable size window using this:
time for i in {0..99}; do screencapture -l 56 -t jpg fred-$i.jpg; done
where I got the 56 from the windowlist program in my other, linked answer.