I have an nsdocument subclass which handles the loading of a binary file (of an existing format, not of my creation.) The file format uses 3 files with the same name for each logical "document" (with different extensions).
I want to be able to have the user select the 3 files in the NSOpenPanel, and then have one NSDocument subclass with its corresponding window created. I know how to set a panel to allow selection of multiple files, but I don't know how to get to the panel that NSDocumentController is using, or what methods to override for loading binary data out of the multiple files.
Relying on the user to do things some singular correct way to work around your app's own limitations is bound to fail. Whatever you tell the user to do, they will do that, the opposite of that, and everything orthogonal to that.
I'd declare each of the three types in my plist, and route them all to the same document class. The document would read from all three files regardless of which one it had been created with, and return an error if any of them were missing. You should probably appoint one of the three to be the document's primary file, setting your fileURL to the URL to that file.
Take care to implement File Coordination correctly (you should be a presenter for all three files and coordinate any attempts to write to any of them). You'll be putting users at risk of data corruption or loss if you don't.
You should not attempt to mess with the Open panel at all. The Open panel is not the only way to open documents in a document-based app. The things you need to do, you need to do behind your GUI, not in it.
If your app is sandboxed, this will be either tricky or impossible. Security-scoped URL bookmarks may help you here.