Actions on Google app created two different user IDs for the Primary user.
I'm using the same device, same phone and app. The first time, it created an alphanumeric Id. The second time it created numeric Id.
This issue affected our user personalization.
userId isn't a very reliable way for customization as it's subject to change under certain circumstances like when your voice doesn't match on Google Home devices (Your case) or the user resets the app which generates a new userId You can find more about user identity here.
A better approach will be to use account linking.