I would like to extend / override the login mechanism of the FELogin extension. AFAIK the FELogin extension just submit the entered data and the TYPO3 CMS authentication services handle the rest. Does someone know which file I have to tinker with to change the authentification logic?
As you already wrote, felogin does only take care about:
All other stuff happens in TYPO3 core and can be manipulated by writing a custom AuthService. There are a lot of extensions in TER which can be looked at if you need some examples.
For starting I would propose to look at some IP restriction extensions as those are the more simplier ones. The API didn't change much in later versions.
The docs can be found here.