pythongoogle-cloud-platformgoogle-cloud-pythongoogle-cloud-instances
Google cloud how to start service when creating instance using Python api?
I'm having a lot of troubles to create instance and start them using Python Google Cloud's api.
Basically I first created all the VMs and setted up the services. Then once everything was working, I stopped the VMs and created Image so I can easily create instance using the images.
[EDIT] I think it's nice to add that :
- frontend instance: nextjs
- backend instance: nodejs
- database instance: postgresql
Here is my Python script:
import googleapiclient.discovery
import argparse
import os
import time
backend_script = """
#! /bin/bash
cd /mse-tsm-cloudsys-lab1/api/
sudo npm install
sudo npx knex migrate:latest
sudo npx knex seed:run
sudo npm run start
"""
frontend_script = """
#! /bin/bash
cd /mse-tsm-cloudsys-lab1/app/
sudo npm install
sudo npm run build
sudo npm run start
"""
def create_instance(compute, project, zone, name, image, machine, ip, ip_public, tags=[], metadata=[]):
config = {
'name': name,
'machineType': "zones/%s/machineTypes/%s" % (zone, machine),
'disks': [
{
'initializeParams': {
"sourceImage": image
},
"boot": True
}
],
'tags': {
"items": tags
},
"networkInterfaces": [
{
'network': 'global/networks/default',
'networkIP': ip,
'accessConfigs': [
{
'type': 'ONE_TO_ONE_NAT',
'name': 'External NAT',
'natIP': ip_public
}
]
}
],
"metadata": {
"items": metadata
}
}
return compute.instances().insert(
project=project,
zone=zone,
body=config).execute()
def delete_instance(compute, project, zone, name):
return compute.instances().delete(
project=project,
zone=zone,
instance=name).execute()
def list_instances(compute, project, zone):
result = compute.instances().list(project=project, zone=zone).execute()
return result['items'] if 'items' in result else None
def wait_for_operation(compute, project, zone, operation):
print('Waiting for operation to finish...')
while True:
result = compute.zoneOperations().get(
project=project,
zone=zone,
operation=operation).execute()
if result['status'] == 'DONE':
print("done.")
if 'error' in result:
raise Exception(result['error'])
return result
time.sleep(1)
def main(mode, project, zone, wait=True):
compute = googleapiclient.discovery.build('compute', 'v1')
if(mode == 'list'):
instances = list_instances(compute, project, zone)
print('Instances in project %s and zone %s:' % (project, zone))
for instance in instances:
print(' - ' + instance['name'])
return
print("Creating Database")
instance_name = "postgres-lab1"
image = "projects/tsm-cloudsys-vial/global/images/image-postgres-lab1"
machine = "e2-micro"
ip = "10.132.0.2"
ip_public = "34.79.195.77"
tags = ["postgres-lab1"]
operation1 = create_instance(compute, project, zone, instance_name, image, machine, ip, ip_public)
print("Creating Backend")
instance_name = "backend-lab1"
image = "projects/tsm-cloudsys-vial/global/images/image-backend-lab1"
machine = "e2-micro"
ip = "10.132.0.4"
ip_public = "34.79.111.217"
metadata = [{
"key": "startup-script",
"value": backend_script
}]
tags = ["backend-lab1","http-server","https-server"]
operation2 = create_instance(compute, project, zone, instance_name, image, machine, ip, ip_public, metadata=metadata)
print("Creating Frontend")
instance_name = "frontend-lab1"
image = "projects/tsm-cloudsys-vial/global/images/image-frontend-lab1"
machine = "e2-micro"
ip = "10.132.0.3"
ip_public = "35.187.109.153"
metadata = [{
"key": "startup-script",
"value": frontend_script
}]
tags = ["frontend-lab1","http-server","https-server"]
operation3 = create_instance(compute, project, zone, instance_name, image, machine, ip, ip_public, tags, metadata=metadata)
print("Waiting for VMs to come online")
wait_for_operation(compute, project, zone, operation1['name'])
wait_for_operation(compute, project, zone, operation2['name'])
wait_for_operation(compute, project, zone, operation3['name'])
instances = list_instances(compute, project, zone)
print('Instances in project %s and zone %s:' % (project, zone))
for instance in instances:
print(' - ' + instance['name'])
if wait:
input()
print("Deleting DB")
operation1 = delete_instance(compute, project, zone, "postgres-lab1")
print("Deleting Backend")
operation2 = delete_instance(compute, project, zone, "backend-lab1")
print("Deleting Frontend")
operation3 = delete_instance(compute, project, zone, "frontend-lab1")
wait_for_operation(compute, project, zone, operation1['name'])
wait_for_operation(compute, project, zone, operation2['name'])
wait_for_operation(compute, project, zone, operation3['name'])
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument('--mode', help='Wanted mode', default='list')
parser.add_argument('project_id', help='Your Google Cloud project ID.')
parser.add_argument(
'--zone',
default='europe-west1-b',
help='Compute Engine zone to deploy to.')
args = parser.parse_args()
main(args.mode, args.project_id, args.zone)
The 3 instances are created and running, I can ping all of them. (yes one is shutted down because I'm trying to reboot it)
But I have two main issues :
- It's impossible to connect to an instance using SSH on the Google Cloud dashboard, it takes forever...
- I feel like my
startup-scriptaren't corrects because even if the 3 instances are running, I can't access them like I could before when I created them manually. For examplefrontend-lab1should be responding on port 3000 but it isn't when I create it using my script
My main concern is the issue number 2 because I think it'll fix the issue number 1. Can someone help me with my startup-script ?
[EDIT2] Here is the log:
ct 6 19:24:23 frontend-lab1 systemd[1]: Starting Google Compute Engine Startup Scripts...
Oct 6 19:24:23 frontend-lab1 google_metadata_script_runner[795]: 2021/10/06 19:24:23 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct 6 19:24:37 frontend-lab1 systemd[1]: systemd-fsckd.service: Succeeded.
Oct 6 19:24:51 frontend-lab1 systemd[1]: systemd-hostnamed.service: Succeeded.
Oct 6 19:25:21 frontend-lab1 GCEGuestAgent[539]: 2021-10-06T19:25:21.4654Z GCEGuestAgent Info: Removing user jeromevialhes.
Oct 6 19:25:22 frontend-lab1 google_guest_agent[539]: 2021/10/06 19:25:22 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct 6 19:25:22 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:22 GCEMetadataScripts: startup-script: /tmp/metadata-scripts111777770/startup-script: line 3: cd: /mse-tsm-cloudsys-lab1/app/: No such file or directory
Oct 6 19:25:23 frontend-lab1 google_metadata_script_runner[795]: 2021/10/06 19:25:23 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct 6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN saveError ENOENT: no such file or directory, open '/package.json'
Oct 6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN enoent ENOENT: no such file or directory, open '/package.json'
Oct 6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No description
Oct 6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No repository field.
Oct 6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No README data
Oct 6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No license field.
Oct 6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script:
Oct 6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: up to date in 0.489s
Oct 6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: found 0 vulnerabilities
Oct 6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script:
Oct 6 19:25:27 frontend-lab1 google_metadata_script_runner[795]: 2021/10/06 19:25:27 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct 6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! code ENOENT
Oct 6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! syscall open
Oct 6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! path /package.json
Oct 6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! errno -2
Oct 6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! enoent ENOENT: no such file or directory, open '/package.json'
Oct 6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! enoent This is related to npm not being able to find a file.
Oct 6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! enoent
Oct 6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script:
Oct 6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! A complete log of this run can be found in:
Oct 6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! /root/.npm/_logs/2021-10-06T19_25_28_453Z-debug.log
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! code ENOENT
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! syscall open
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! path /package.json
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! errno -2
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! enoent ENOENT: no such file or directory, open '/package.json'
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! enoent This is related to npm not being able to find a file.
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! enoent
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script:
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! A complete log of this run can be found in:
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! /root/.npm/_logs/2021-10-06T19_25_29_171Z-debug.log
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script exit status 254
Oct 6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: Finished running startup scripts.
Oct 6 19:25:29 frontend-lab1 systemd[1]: google-startup-scripts.service: Succeeded.
Oct 6 19:25:29 frontend-lab1 systemd[1]: Finished Google Compute Engine Startup Scripts.
But you can see that the repository exists :
jeromevialhes@frontend-lab1:~$ ls
mse-tsm-cloudsys-lab1
Solution
Looking at the logs you included in the question, it looks like there could be a problem with your custom image. In line 7 of the log file, it warns that the directory “/mse-tsm-cloudsys-labl1/app/” is not found:
Oct 6 19:25:22 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:22 GCEMetadataScripts: startup-script: /tmp/metadata-scripts111777770/startup-script: line 3: cd: /mse-tsm-cloudsys-lab1/app/: No such file or directory
If this directory was used to contain your application files, it would be the cause of the subsequent errors related to NPM not being able to find files such as package.json and other files that belong to your application.
You should confirm whether the application path is correct or if the image was created correctly. You can follow this guide about creating custom images from your existing VM instances, and use the Node.js and Python guides to make sure you are following the correct steps.
- Is it possible to call setter as a method?
- Accessing a Python traceback from the C API
- Scraping hierarchical website in a specific category
- How to offset a marker in matplotlib?
- How to access current user in Django class based view
- Is there a way to define list(obj) method on a user defined class in python?
- Deserializing Json list into class objects not able to extract attrributes
- What is the pythonic way to detect the last element in a 'for' loop?
- How to get integer values from a string in Python?
- Optimize this Python code that involves matrix inversion
- how to read a file with dictionary?
- How to extract only audio from downloading video? Python yt-dlp
- How to solve Invalid element(s) received for the 'color' property?
- Plotly Value error - Invalid property for colour
- no module named distutils....but distutils installed?
- how can i change the venv of my current shell from within a python script
- What is the difference between "cast(...)" and "... # type: ..."?
- getting the file size with pycurl
- export conda environment without prefix variable which shows local path to executable
- How can I fix datetime error when trying to use the now syntax?
- Python 3 asyncio with aioboto3 seems sequential
- How can I set OpenCV to use origin as bottom-left corner of image?
- Can I use sleep in the main thread without blocking other threads?
- Getting number of elements in an iterator in Python
- Merging tuples if they have one common element
- How to specify the response_model in FastAPI on a non-default return?
- PyCharm doesn’t see files in project
- How to tune PID controller on non-linear model (here: diving of a body)
- Create list of factorials using list comprehension
- Rewriting URL request function to satisfy GitHub CodeQL server side request forgery (SSRF) warning