So I've installed msysgit, and run git successfuly from the bash shell.
From within the bash shell, I've created keys with ssh-keygen - everything works fine.
Key files are in my %userprofile%\.ssh folder (from the bash shell, it is: ~/.ssh).
I'm able to connect to my server (unfuddle in my case).
Next, I do want to be able to run git from the Windows Cmd shell.
Adding c:\msysgit\bin and c:\msysgit\mingw\bin to the search path.
Git is working locally just fine: branch, merge, add, commit, checkout, ... working well.
However, trying to work with the server fails, with: 'Permission denied (publickey).'
It looks like git, when executed from command line looks at a different location trying to allocated the key files, or another reason causing it to fail.
To get the mysysgit command line to find the .ssh keys you'll need to have %HOME% point to the directory containing the .ssh folder
Often you can just do
set HOME=%USERPROFILE%