I am using very aggressive video compression, eg -crf 51. I am using this for 'artistic' effect, so what I am doing may not make sense from a normal video compression point of view.
So far I have only been using very basic compression control using only the -crf or -b:v flags. The results look like ffmpeg divides images into square patches and the makes smooth approximations within the patches. This gives 2 control dimensions to the process: the patch size and the aggressiveness of the smoothing within the patches.
It have found that ffmpeg uses both parameters to some extent, but there appears to be an absolute maximum patch size in pixels beyond which it will not go regardless of the frame size. After that it will only increase compression by reducing the detail within the patches.
This is suboptimal for high resolution video, where this becomes equivalent to reducing the resolution. The problem is particularly noticeable on fractal like images which have large featureless region as well as regions of high detail.
How can I tell ffmpeg to increase the maximum patch size and retain more detail within the patches?
What you are calling a 'patch' is a macroblock. Every codec has a maximum macroblock size, and it is generally very small (16x16 pixles). It is a fundamental property of how this type of compression works and can not be set to arbitrary values. You will need to create your effect another way.