I want to close running instance of my application on repair option of my setup....
This can be done through a custom action. There is no predefined support for it, so you will need to write your own custom code. A good approach is sending WM_CLOSE to the main application window and let it close itself.
To run this custom action only during repair, you can try conditioning it with REINSTALL property.
If you want to avoid the Files In Use dialog, you will need to run this custom action before InstallValidate action. In this case you cannot use REINSTALL property, but you can use this condition:
Installed
It evaluates to true when your package is running in maintenance mode (modify, repair or remove).