I read many blogs and post in Stackoverflow but could not understand exactly which one is appropriate in which situation.
What I understood till now is, custom authentication handler should be written when user needs to redirected to 3rd party system for authentication and then AuthenticationInfo object is sent to the DefaultLogin module.
Now custom login module is used when there is a need to sync user data into AEM from 3rd Party system. During the synchronization process custom login module also authenticate user against 3rd party. But this can also be possible in authentication handler also.
If I look at the out of the box SAML authentication handler then it does not have login module to synchronize user data, rather SAML authentication handler itself synchronize user data. Why there is such difference in implementation? Which one is applicable in which scenario? Does login module gives extra level of security?
Please note that Login module has been rewritten and now its call External Identity Provider.
External identity provider does not only sync user data but also authenticate user entered credentials. Lets take an example where you need to authenticate user against 3rd party system (which means you need to ask user to enter username and password through Authentication handler in extracthandler method) and after user enters his/her credentials then you want that credentials to be validated again before granting permission to the repository (in this case you need to write External Identity Provider). One example could be, once user enter credentials then 3rd party system generates some token. Now you can validate this token in your External Identity Provider code by calling some web service endpoint provided by 3rd party.
More details here