Search code examples
google-cloud-platformnlpgoogle-compute-enginegoogle-cloud-sdk

ERROR: (gcloud.compute.instances.create) Could not fetch resource: - Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 0.0 globally

I would like to try PEGASUS to summarize article. https://github.com/google-research/pegasus

I followed this instruction. https://github.com/google-research/pegasus/tree/f76b63c2886748f7f5c6c9fb547456d8c6002562#setup

I checked the region which I can use NVIDIA Tesla V100 and I decided to use us-central1-a https://cloud.google.com/compute/docs/gpus

I used this command.

gcloud compute instances create pegasustest --zone=us-central1-a 
--machine-type=n1-highmem-8 --accelerator type=nvidia-tesla-v100,count=1 
--boot-disk-size=500GB --image-project=ml-images --image-family=tf-1-15 
--maintenance-policy TERMINATE --restart-on-failure

I got this error message.

ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/covid19agent/zones/us-central1-a' does not have enough 
resources available to fulfill the request.  
Try a different zone, or try again later.

I took 3 hours and tried again, but I got the same result.

So, I changed the region from us-central1-a to asia-east1-c.

I used this command.

gcloud compute instances create pegasustest --zone=asia-east1-c 
--machine-type=n1-highmem-8 --accelerator type=nvidia-tesla-v100,count=1 
--boot-disk-size=500GB --image-project=ml-images --image-family=tf-1-15 
--maintenance-policy TERMINATE --restart-on-failure

Then I got this error message.

WARNING: Some requests generated warnings:
 - Disk size: '500 GB' is larger than image size: '10 GB'. 
You might need to resize the root repartition manually 
if the operating system does not support automatic resizing. 
See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd 
for details.

ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'GPUS_ALL_REGIONS' exceeded.  Limit: 0.0 globally.

Is it impossible for me to try PEGASUS? And, does it cost too much to try PEGASUS?

Solution

  • Let's start with the first issue. Have a look again at the error message:

    ERROR: (gcloud.compute.instances.create) Could not fetch resource: - The zone 'projects/covid19agent/zones/us-central1-a' does not have enough resources available to fulfill the request. Try a different zone, or try again later.

    When you start an instance it requests resources like vCPU, memory, GPU and if there's not enough resources available in the zone you'll get such message, more information available in the documentation:

    If you receive a resource error (such as ZONE_RESOURCE_POOL_EXHAUSTED or ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS) when requesting new resources, it means that the zone cannot currently accommodate your request. This error is due to Compute Engine resource obtainability, and is not due to your Compute Engine quota.

    Resource availability are depending from users requests and therefore are dynamic.

    There are a few ways to solve this issue:

    1. Wait for a while and try to start your VM instance again (as you tried, but fruitless this time).
    2. Move your instance to another zone (as you did).

    3. Reserve resources for your VM by following documentation to avoid such issue in future:

      Create reservations for Virtual Machine (VM) instances in a specific zone, using custom or predefined machine types, with or without additional GPUs or local SSDs, to ensure resources are available for your workloads when you need them. After you create a reservation, you begin paying for the reserved resources immediately, and they remain available for your project to use indefinitely, until the reservation is deleted.

    Now, let's have a look at the second issue. Have a look again at this error message:

    ERROR: (gcloud.compute.instances.create) Could not fetch resource: - Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 0.0 globally.

    More information about quotas you can find in the documentation.

    To solve this issue you should follow steps below:

    1. Ensure that billing is enabled for your project.

    2. Request an increase in quota:

      1. Go to the Quotas page.
      2. In the Quotas page, select the quotas you want to change.
      3. Click the Edit Quotas button on the top of the page.
      4. Check the box of the service you want to edit.
      5. Fill out your name, email, and phone number, and click Next.
      6. Enter your request to increase your quota, and click Next.
      7. Submit your request.
      8. A request to decrease quota is rejected by default. If you must reduce your quota, reply to the support email with an explanation of your requirements. A support representative from the Compute Engine team will respond to your request within 24 to 48 hours.

      You're not able to request an increase in quota if you use 12-month, $300 free trial because of the limitations:

      Your free trial credit applies to all Google Cloud resources, with the following exceptions:

      • You can't have more than 8 cores (or virtual CPUs) running at the same time.
      • You can't add GPUs to your VM instances.
      • You can't request a quota increase. For an overview of Compute Engine quotas, see Resource quotas.
      • You can't create VM instances that are based on Windows Server images.

      You must upgrade your account to perform any of the actions in the preceding list.

    You can estimate cost of usage with Google Cloud Pricing Calculator.