I want to have VM instance that contains installed libraries/dependencies accessible for different users. I grant the roles like Compute Admin, Service Account User etc. for the users accordingly. Although they can see the folders/files/codes from the shared home folder, they can't use the installed libraries and other drivers for instance.
Every user has to access with his SSH key and I don't want them to install packages and drivers from scratch. Is it possible to accomplish?
You could modify the user roles for those to have compute login on the instance and follow the instructions on setting up os-login to allow them to SSH via gcloud (using their own accounts).
You can create a custom OS image based on one of the supported images that you use and bake the dependencies into that custom image.
Or
Use a startup script to install/grab dependencies at startup.
As John said in the comments, you want to ensure your users don't have SUDO privilege as they could impact others.