Whenever anyone uses torrent magnetic link of torrent there is a pop up that comes up with external protocol request title. I want to know how this protocol works and how this can open an application on client from a web page.
The dialog you're referring to is completely handled by the browser that you're using and the protocol of the link that you're clicking.
Normally, links between web pages uses the following scheme:
http://www.website.com/something
If you change the protocol (in this example the "http" part) you will then refer to another application, if installed in the client machine. For example, if you click on a link that point to
ftp://www.website.com/something
your browser will search for an application that can handle the "ftp" protocol.
In general, when a browser detects a link that is not something that it knows, it tries to redirect the request to another application that has been previously installed by the user and that has stated some sort of compatibility with that protocol.
The popup is not something related to the magnet link. If you'd like to get further information on the protocol, this Wikipedia page explains it clearly.