I have TextMate running on Mac, and it has txmt protocol, that I can use as on web browser such as txmt://open/?url=file:///Users/smcho/smcho/bin/rst2html. I can use TextMate for editing rst2html.
However, when I tried to use this protocol with Org-mode as [txmt://open/?url=file:///Users/smcho/smcho/bin/rst2html], instead of opening TextMate, the rst2html binary is running.
I tried
(setq org-link-abbrev-alist
'(
("edit" . "txmt://open/?url=file://%s")
))
to use [edit:/Users/smcho/smcho/bin/rst2html], but I get the same result.
How can I use specific protocol (in my case : txmt) with Org mode?
It's not using protocol, but one can run open and edit some file with this command.
("edit" . "shell:/usr/local/bin/mate %s")
I believe org-mode can only handle links specified on this page (and maybe a few other link types, e.g. links to git repos, courtesy of contrib packages). I'm not a TextMate/Apple user, but it seems to me that txmt:// protocol in the question is just a handler TextMate sets up for the installed browsers; you can't expect this to work with org.
The real question is -- can you set up a link so that a particular file is opened in TextMate. The most straightforward way is to issue a shell command that will open the file in TextMate. Using org-link-abbrev-alist, you could do something like ("edit" . "shell:/path/to/textmate_binary %s &"). You'll also want to use (setq org-confirm-shell-link-function nil) to suppress the confirmation prompt.
In general, it is always possible to create custom link types as described here; this involves writing the handler function in elisp. In this case, this would be a call to start TextMate using e.g. start-process. This would be a cleaner solution, but probably not worth the extra effort. In customizing org link handlers, it is usually sufficient to set up shortcuts via org-link-abbrev-alist, or set up handlers based on extension via org-file-apps.