I want to define a custom file format that my application can recognize. The file will be effectively an archive but I want the file extension to be different and also with a different mimetype. The content of the archive will be in a specific structure so I can easily find the entries.
Also, when a user is uploading, I want the browser to filter the directory and only show only files with this extension.
What is the best approach to address this problem.
It depends mainly on what you mean by "recognise".
For instance, in the general sense, you don't need to do much work at all. If you define something called *.qip which has the same general structure as a ZIP file, then all you have to do is have your application read that file as if it were a standard zip file. The file name (and to a lesser extent, MIME type) have no bearing at all on how your application handles the file - it will do so exactly as you've coded it to do so.
Now if you've coded your application to use a third-party library to do this file handling, you'd need to look at the library docs as this will vary as to how to achieve it (if it's supported at all).
As for the file upload form field, that's trickier. Theoretically you can specify a MIME type filter using the accept attribute, but that's ignored by I think just about every major browser out there. In light of that, I don't think there's anything robust that you can do client-side to check this - but hey, at least no-one else can do this either. And you are checking the validity of the files on the server-side, right? :-)