Is there a recommended/common way for showing an application-modal "New Document" window when the user chooses File > New or presses Cmd+N?
I am talking about a window where the user can set initial document properties like page dimensions. There should be both OK and Cancel buttons, which means that dismissing the New Document window will not necessarily lead to an actual document window being created.
By the way, my application is NSDocument-based.
What you're asking is fairly straightforward. Create a window and window controller for the new-document dialog. Have your app delegate instantiate them on startup or on demand. Set Cmd-N as the key equivalent for your File > New item, and connect it to an action in your app delegate which sets up and runs a modal session for that window ([NSApp runModalForWindow:wc.window]).
The window controller should end or cancel the modal session, decide whether or not to create the document (probably using [-NSDocumentController openUntitledDocumentAndDisplay:error:]), and set the attributes you need to set.
That said, I urge you not to make it application-modal. Open a new document window and present a document-modal sheet which asks for and sets those dimensions instead. No reason to stop the user from doing whatever else s/he might be doing in order to answer those questions.