Popup asking for credentials on git-lfs repeateadly

I have very annoying popups that asks for password every time I try to push to a remote repo with git lfs enabled. The popups area simple "Enter passphrase for key 'name_of_the_SSH_key'. The title is "Git for Windows". The commits are often still pushed if pressing cancel repeatedly (if no git-lfs objects pushed).

How to prevent that ?

More context :

I use SSH, on Windows 11, git version : git version 2.41.0.windows.1.
That popups happen on every project.
I tried using manager and cache in credential gitconfig.
Here is my git config (following a few post suggestions, without success):

[credential]
  helper =
  helper = cache
  guiPrompt = false
  modalPrompt = false

Result from the command git gc && GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push, before the popup called

10:15:21.819070 exec-cmd.c:243          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
10:15:21.831913 git.c:462               trace: built-in: git push
10:15:21.837497 run-command.c:661       trace: run_command: unset GIT_PREFIX; ssh git@gitlab.com 'git-receive-pack 'repo_address'
Enter passphrase for key '/c/Users/username/.ssh/id_ed25519': 
10:15:26.925370 run-command.c:661       trace: run_command: .git/hooks/pre-push origin git@gitlab.com:repo_address.git
10:15:27.026482 exec-cmd.c:243          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
10:15:27.039931 git.c:748               trace: exec: git-lfs pre-push origin git@gitlab.com:repo_address.git
10:15:27.039931 run-command.c:661       trace: run_command: git-lfs pre-push origin git@gitlab.com:repo_address.git
10:15:27.066754 trace git-lfs: exec: git 'version'
10:15:27.142308 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote' '-v'
10:15:27.202861 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
10:15:27.249710 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
10:15:27.294826 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
10:15:27.344745 trace git-lfs: exec: uname 
10:15:27.419941 trace git-lfs: exec: git 'config' '--includes' '-l'
10:15:27.465155 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
10:15:27.511599 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
10:15:27.558154 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
10:15:27.604909 trace git-lfs: attempting pure SSH protocol connection
10:15:27.605412 trace git-lfs: run_command: ssh -oControlMaster=auto -oControlPath=C:\Users\USERNAME\AppData\Local\Temp\sock-2312666405\sock-%C git@gitlab.com git-lfs-transfer .git upload
10:15:27.609163 trace git-lfs: exec: ssh '-oControlMaster=auto' '-oControlPath=C:\Users\USERNAME\AppData\Local\Temp\sock-2312666405\sock-%C' 'git@gitlab.com' 'git-lfs-transfer repo_address.git upload'

Thanks for any help or tip !

Solution

Well, I finally found the solution to this : It seems my credential manager was not set up properly (sigh...).

Following the help from github : https://docs.github.com/en/authentication/connecting-to-github-with-ssh/working-with-ssh-key-passphrases

Adding the ssh-agent solved the issue.