By default, GeoServer uses weak password-based encryption for the default usergroup and shows a hint in the web administration interface to change it to Hash (Digest). This can be done in the interface under Users, Groups, Roles > User Group Services > default > Passwords.
I would like to change the password encryption via REST. Is that possible? I haven't found any matching endpoint in the documentation.
I don't believe that functionality is provided via the REST API. If you would like to add it we would welcome a PR adding the functionality.