How can I restrict the access of certain users to "retrieve secret value" from the secret manager or to give access only to certain users? How could I do this from boto3 or even from the console.
The AWS docs have a tutorial on ABAC (Attribute Based Access Control; a.k.a tagging) that describes how to do this. The tutorial is a bit of a long read though, and to sum it up, you can put a tag on the secrets and tag on your users/roles and compare the tag values in the IAM policies with the condition:
"Condition": {
"StringEquals": {
"aws:ResourceTag/MYTAGNAME": "${aws:PrincipalTag/MYTAGNAME}"
}
},
You can also do the same using resource policies on the secrets.