Search code examples
pythontorchgoogle-cloud-dataproc

import torch not defined on gcp

Trying to run "import torch" on GCP.

I got the message ModuleNotFoundError: No module named 'torch'

Tried to create a new cluster (through the api, not terminal) with the following properties:

dataproc->conda.packages->pytorch==1.7.1,visions==0.7.1

dataproc->pip.packages->tokenizers==0.10.1,datasets==1.5.0

The creation of the cluster timed out and was not created. the same happened even when tried just

dataproc -> conda.packages -> pytorch==1.7.1

Trying to conda install on the master manually with the following :

conda install -c pytorch pytorch

After more almost an hour, I got :

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment -/

Thanks for any help or advice i.e., except torch I also need : from transformers import LongformerModel, LongformerTokenizer from model import RedditModel

Solution

  • You just need to use a newer/latest PyTorch version available in conda-forge channel (1.8.0 now):

    gcloud dataproc clusters create ${CLUSTER_NAME} \
  --image-version=2.0 \
  --region=${REGION} \
  --properties='^#^dataproc:conda.packages=pytorch==1.8.0,torchvision==0.9.0'