Search code examples
pythongoogle-cloud-platformgcloud

"gcloud storage cp t.txt gs://my.bucket.name/" gives error: ModuleNotFoundError: No module named 'google.auth'

I'm trying to upload a file to a bucket using:

gcloud storage cp t.txt gs://my.bucket.name/

But there's an error:

Copying file://t.txt to gs://adhoc.textra.me/t.txt
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Cellar/[email protected]/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/share/google-cloud-sdk/lib/googlecloudsdk/command_lib/storage/tasks/task_graph_executor.py", line 37, in <module>
    from googlecloudsdk.command_lib.storage import encryption_util
  File "/opt/homebrew/share/google-cloud-sdk/lib/googlecloudsdk/command_lib/storage/encryption_util.py", line 28, in <module>
    from googlecloudsdk.command_lib.storage import hash_util
  File "/opt/homebrew/share/google-cloud-sdk/lib/googlecloudsdk/command_lib/storage/hash_util.py", line 25, in <module>
    from googlecloudsdk.command_lib.storage import fast_crc32c_util
  File "/opt/homebrew/share/google-cloud-sdk/lib/googlecloudsdk/command_lib/storage/fast_crc32c_util.py", line 32, in <module>
    from googlecloudsdk.command_lib import info_holder
  File "/opt/homebrew/share/google-cloud-sdk/lib/googlecloudsdk/command_lib/info_holder.py", line 45, in <module>
    from googlecloudsdk.core.credentials import store as c_store
  File "/opt/homebrew/share/google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py", line 34, in <module>
    from googlecloudsdk.api_lib.auth import external_account as auth_external_account
  File "/opt/homebrew/share/google-cloud-sdk/lib/googlecloudsdk/api_lib/auth/external_account.py", line 24, in <module>
    from googlecloudsdk.core.credentials import creds as c_creds
  File "/opt/homebrew/share/google-cloud-sdk/lib/googlecloudsdk/core/credentials/creds.py", line 33, in <module>
    from google.auth import compute_engine as google_auth_compute_engine
ModuleNotFoundError: No module named 'google.auth'
  Completed files 0/1 | 0B/16.5kiB

It hangs at that point and doesn't upload.

This is on Macos with the google-cloud-sdk installed via Brew:

$ brew info google-cloud-sdk
==> google-cloud-sdk: 479.0.0 (auto_updates)
https://cloud.google.com/sdk/
Installed
/opt/homebrew/Caskroom/google-cloud-sdk/479.0.0 (132B)
From: https://github.com/Homebrew/homebrew-cask/blob/HEAD/Casks/g/google-cloud-sdk.rb
==> Name
Google Cloud SDK
==> Description
Set of tools to manage resources and applications hosted on Google Cloud
==> Artifacts
google-cloud-sdk/install.sh (Installer)
google-cloud-sdk/bin/gsutil (Binary)
google-cloud-sdk/completion.bash.inc -> /opt/homebrew/etc/bash_completion.d/google-cloud-sdk (Binary)
google-cloud-sdk/bin/bq (Binary)
google-cloud-sdk/bin/docker-credential-gcloud (Binary)
google-cloud-sdk/completion.zsh.inc -> /opt/homebrew/share/zsh/site-functions/_google_cloud_sdk (Binary)
google-cloud-sdk/bin/gcloud (Binary)
google-cloud-sdk/bin/git-credential-gcloud.sh -> git-credential-gcloud (Binary)
==> Caveats
To add gcloud components to your PATH, add this to your profile:

  for bash users
    source "$(brew --prefix)/share/google-cloud-sdk/path.bash.inc"

  for zsh users
    source "$(brew --prefix)/share/google-cloud-sdk/path.zsh.inc"
    source "$(brew --prefix)/share/google-cloud-sdk/completion.zsh.inc"

  for fish users
    source "$(brew --prefix)/share/google-cloud-sdk/path.fish.inc"

==> Analytics
install: 11,361 (30 days), 33,066 (90 days), 118,204 (365 days)

The SDK has been updated to the latest version:

$
~/delicious/textra (git)  $ gcloud -v
Google Cloud SDK 487.0.0
beta 2024.08.06
bq 2.1.7
core 2024.08.06
gcloud-crc32c 1.0.0
gsutil 5.30

Uploading using gsutil works fine, but I want to use gcloud compute cp ... so I can specify a configuration, e.g. gcloud --configuration=dev compute cp ....

Edit: I've tried completely uninstalling and deleting config files and reinstalling, with the same result.

Edit: it turns out unsetting the CLOUDSDK_PYTHON_SITEPACKAGES environment variable fixed the issue!

Solution

  • First, check your .bash_profile to ensure there's nothing that could interfere with any CLOUDSDK_PYTHON variables.

    If not, you should also check that :

    Your command brew info google-cloud-sdk is returning a Caveat :

    ==> Caveats
To add gcloud components to your PATH, add this to your profile:

  for bash users
    source "$(brew --prefix)/share/google-cloud-sdk/path.bash.inc"

  for zsh users
    source "$(brew --prefix)/share/google-cloud-sdk/path.zsh.inc"
    source "$(brew --prefix)/share/google-cloud-sdk/completion.zsh.inc"

  for fish users
    source "$(brew --prefix)/share/google-cloud-sdk/path.fish.inc"

    You should run the one that match your shell and add it to your bash profile as mentioned.

    If it doesn't work, try : gcloud components reinstall

    If it still doesn't work, you can check the Python used by gcloud using : gcloud info --format='value(basic.python_location)'

    Take the returned path, replace python to pip and add install google-auth, then run the command. It's going to use the right Python/Pip to install google.auth.

    As a last resort, consider reinstalling gcloud by following the GCP instructions.