Following the answer here:
Set QuickLook window size when previewing with QLPreviewingController.preparePreviewOfFile I set the preferredContentSize of the QLPreviewingController, which changes the window size as intended, but this seems to break subsequent auto resizing of all subviews. After setting it, the content will no longer track the preview window size when the user resizes it manually or enters fullscreen.
Any attempts to set view sizes, resizing masks or constraints programmatically after setting the preferredContentSize property will cause errors like
<NSViewServiceMarshal: 0x7fe1b740f170 <QLPreviewExtensionViewController: 0x600000e31b00 <NSServiceViewControllerWindow: 0x7fe1b760b350>>> attempted to set frame of <NSServiceViewControllerWindow: 0x7fe1b760b350> from {{17, 382}, {820, 846}} to {{17, 382}, {800, 600}} and got {{17, 136}, {820, 846}}; set breakpoint on -[NSViewServiceMarshal actualFrame:differsFromSpecifiedFrame:oldFrame:ofWindow:] to catch this
and have no visible effect. Setting the breakpoint only tells me that it happens in a system background thread at about the time I try to change the size of the view.
EDIT: Another finding: If I print the frames values of all views and the window to the log, the printed size is the one I requested, not the smaller one actually showing on the screen! I.e. if I set preferredContentSize to (820, 846), my debug NSLogs and the Xcode Debug View Hierarchy shows a window that is 820 x 846, while the actual window showing onscreen is twice as wide as it is high.
EDIT: Made a video to illustrate this problem: https://youtu.be/l7b1jq7pDWY
Original code is here: https://github.com/angstsmurf/spatterlight/blob/quicklook-ifiction/iFictionQuickLook/iFictionPreviewController.m
This seems to be a Catalina only bug (earlier versions do not support this kind of Quicklook extension.) Upgrading to Big Sur made it go away.