It seems to show this dialog two or three times. If I cancel it goes away and I'm still able to commit changes.
How do I prevent it from popping up each time and where do I find my passphrase if I have to reenter it and click "Remember to keychain" (which I've done before but will try again)?
In the latest version of macOS - 10.12.2/Sierra - this is an easy fix. Just edit your ~/.ssh/config and enable the UseKeychain option:
Host *
UseKeychain yes
Save and solved.