I would like to access and manage the Drive of a single user via the Google Drive API. I have this working currently, however it does require (or at least seems to require) that the service account have domain-wide authority access and utilize user impersonation.
However, this gives excessive rights to this service account, ultimately it can impersonate any user. Is there any method of restricting the authority of the service account or API access to a single or group of users?
As per the official google documentation, there is no way to do this. In my opinion, this is a feature request. It is similar to the one filed here. Maybe you can submit a new feature request that better explains your intention; however, being the fact that it is a feature request, don't expect something happening soon.
If you don't want user data to be exposed to a client credential app, then perhaps you should use the 3-Legged Auth instead. Have the user sign in to your application, cache the token in a database or somewhere and use the refresh token to refresh the access token when necessary. That way you will ensure that the application has access to read data of the users that it is intended for.