0

I'm having a lot of troubles to create instance and start them using Python Google Cloud's api.

Basically I first created all the VMs and setted up the services. Then once everything was working, I stopped the VMs and created Image so I can easily create instance using the images.

[EDIT] I think it's nice to add that :

  • frontend instance: nextjs
  • backend instance: nodejs
  • database instance: postgresql

Here is my Python script:

import googleapiclient.discovery
import argparse
import os
import time

backend_script = """
#! /bin/bash
cd /mse-tsm-cloudsys-lab1/api/
sudo npm install
sudo npx knex migrate:latest
sudo npx knex seed:run
sudo npm run start
"""

frontend_script = """
#! /bin/bash
cd /mse-tsm-cloudsys-lab1/app/
sudo npm install
sudo npm run build
sudo npm run start
"""

def create_instance(compute, project, zone, name, image, machine, ip, ip_public, tags=[], metadata=[]):
    config = {
        'name': name,
        'machineType': "zones/%s/machineTypes/%s" % (zone, machine),
        'disks': [
            {
                'initializeParams': {
                    "sourceImage": image
                },
                "boot": True
            }
        ],
        'tags': {
            "items": tags
        },
        "networkInterfaces": [
            {
                'network': 'global/networks/default',
                'networkIP': ip,
                'accessConfigs': [
                    {
                        'type': 'ONE_TO_ONE_NAT',
                        'name': 'External NAT',
                        'natIP': ip_public
                    }
                ]
            }
        ],
        "metadata": {
            "items": metadata
        }
    }
    return compute.instances().insert(
        project=project,
        zone=zone,
        body=config).execute()

def delete_instance(compute, project, zone, name):
    return compute.instances().delete(
        project=project,
        zone=zone,
        instance=name).execute()

def list_instances(compute, project, zone):
    result = compute.instances().list(project=project, zone=zone).execute()
    return result['items'] if 'items' in result else None

def wait_for_operation(compute, project, zone, operation):
    print('Waiting for operation to finish...')
    while True:
        result = compute.zoneOperations().get(
            project=project,
            zone=zone,
            operation=operation).execute()

        if result['status'] == 'DONE':
            print("done.")
            if 'error' in result:
                raise Exception(result['error'])
            return result

        time.sleep(1)

def main(mode, project, zone, wait=True):
    compute = googleapiclient.discovery.build('compute', 'v1')

    if(mode == 'list'):
        instances = list_instances(compute, project, zone)

        print('Instances in project %s and zone %s:' % (project, zone))
        for instance in instances:
            print(' - ' + instance['name'])

        return

    print("Creating Database")
    instance_name = "postgres-lab1"
    image = "projects/tsm-cloudsys-vial/global/images/image-postgres-lab1"
    machine = "e2-micro"
    ip = "10.132.0.2"
    ip_public = "34.79.195.77"
    tags = ["postgres-lab1"]
    operation1 = create_instance(compute, project, zone, instance_name, image, machine, ip, ip_public)

    print("Creating Backend")
    instance_name = "backend-lab1"
    image = "projects/tsm-cloudsys-vial/global/images/image-backend-lab1"
    machine = "e2-micro"
    ip = "10.132.0.4"
    ip_public = "34.79.111.217"
    metadata = [{
        "key": "startup-script",
        "value": backend_script
    }]
    tags = ["backend-lab1","http-server","https-server"]
    operation2 = create_instance(compute, project, zone, instance_name, image, machine, ip, ip_public, metadata=metadata)

    print("Creating Frontend")
    instance_name = "frontend-lab1"
    image = "projects/tsm-cloudsys-vial/global/images/image-frontend-lab1"
    machine = "e2-micro"
    ip = "10.132.0.3"
    ip_public = "35.187.109.153"
    metadata = [{
        "key": "startup-script",
        "value": frontend_script
    }]
    tags = ["frontend-lab1","http-server","https-server"]
    operation3 = create_instance(compute, project, zone, instance_name, image, machine, ip, ip_public, tags, metadata=metadata)

    print("Waiting for VMs to come online")
    wait_for_operation(compute, project, zone, operation1['name'])
    wait_for_operation(compute, project, zone, operation2['name'])
    wait_for_operation(compute, project, zone, operation3['name'])

    instances = list_instances(compute, project, zone)

    print('Instances in project %s and zone %s:' % (project, zone))
    for instance in instances:
        print(' - ' + instance['name'])

    if wait:
        input()

    print("Deleting DB")
    operation1 = delete_instance(compute, project, zone, "postgres-lab1")
    print("Deleting Backend")
    operation2 = delete_instance(compute, project, zone, "backend-lab1")
    print("Deleting Frontend")
    operation3 = delete_instance(compute, project, zone, "frontend-lab1")
    wait_for_operation(compute, project, zone, operation1['name'])
    wait_for_operation(compute, project, zone, operation2['name'])
    wait_for_operation(compute, project, zone, operation3['name'])

if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('--mode', help='Wanted mode', default='list')
    parser.add_argument('project_id', help='Your Google Cloud project ID.')
    parser.add_argument(
        '--zone',
        default='europe-west1-b',
        help='Compute Engine zone to deploy to.')

    args = parser.parse_args()

    main(args.mode, args.project_id, args.zone)

The 3 instances are created and running, I can ping all of them. (yes one is shutted down because I'm trying to reboot it) enter image description here But I have two main issues :

  1. It's impossible to connect to an instance using SSH on the Google Cloud dashboard, it takes forever...
  2. I feel like my startup-script aren't corrects because even if the 3 instances are running, I can't access them like I could before when I created them manually. For example frontend-lab1 should be responding on port 3000 but it isn't when I create it using my script

My main concern is the issue number 2 because I think it'll fix the issue number 1. Can someone help me with my startup-script ?

[EDIT2] Here is the log:

ct  6 19:24:23 frontend-lab1 systemd[1]: Starting Google Compute Engine Startup Scripts...
Oct  6 19:24:23 frontend-lab1 google_metadata_script_runner[795]: 2021/10/06 19:24:23 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct  6 19:24:37 frontend-lab1 systemd[1]: systemd-fsckd.service: Succeeded.
Oct  6 19:24:51 frontend-lab1 systemd[1]: systemd-hostnamed.service: Succeeded.
Oct  6 19:25:21 frontend-lab1 GCEGuestAgent[539]: 2021-10-06T19:25:21.4654Z GCEGuestAgent Info: Removing user jeromevialhes.
Oct  6 19:25:22 frontend-lab1 google_guest_agent[539]: 2021/10/06 19:25:22 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct  6 19:25:22 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:22 GCEMetadataScripts: startup-script: /tmp/metadata-scripts111777770/startup-script: line 3: cd: /mse-tsm-cloudsys-lab1/app/: No such file or directory
Oct  6 19:25:23 frontend-lab1 google_metadata_script_runner[795]: 2021/10/06 19:25:23 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN saveError ENOENT: no such file or directory, open '/package.json'
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN enoent ENOENT: no such file or directory, open '/package.json'
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No description
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No repository field.
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No README data
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: npm WARN !invalid#2 No license field.
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: 
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: up to date in 0.489s
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: found 0 vulnerabilities
Oct  6 19:25:27 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:27 GCEMetadataScripts: startup-script: 
Oct  6 19:25:27 frontend-lab1 google_metadata_script_runner[795]: 2021/10/06 19:25:27 logging client: rpc error: code = Unauthenticated desc = transport: metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Flogging.write" not defined
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! code ENOENT
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! syscall open
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! path /package.json
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! errno -2
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! enoent ENOENT: no such file or directory, open '/package.json'
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! enoent This is related to npm not being able to find a file.
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! enoent 
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: 
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR! A complete log of this run can be found in:
Oct  6 19:25:28 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:28 GCEMetadataScripts: startup-script: npm ERR!     /root/.npm/_logs/2021-10-06T19_25_28_453Z-debug.log
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! code ENOENT
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! syscall open
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! path /package.json
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! errno -2
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! enoent ENOENT: no such file or directory, open '/package.json'
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! enoent This is related to npm not being able to find a file.
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! enoent 
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: 
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR! A complete log of this run can be found in:
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script: npm ERR!     /root/.npm/_logs/2021-10-06T19_25_29_171Z-debug.log
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: startup-script exit status 254
Oct  6 19:25:29 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:29 GCEMetadataScripts: Finished running startup scripts.
Oct  6 19:25:29 frontend-lab1 systemd[1]: google-startup-scripts.service: Succeeded.
Oct  6 19:25:29 frontend-lab1 systemd[1]: Finished Google Compute Engine Startup Scripts.

But you can see that the repository exists :

jeromevialhes@frontend-lab1:~$ ls
mse-tsm-cloudsys-lab1
Improve this question
12
  • Did you create a firewall rule on port 22 (SSH) so that you can connect to the instances ? On VPC Network API ? Commented Oct 6, 2021 at 15:14
  • @razimbres yes I created all the rules I needed for all the ports I need (that's why it's working when I create the VM instance manually) but I really think the issue comes from my startup-script Commented Oct 6, 2021 at 15:15
  • This may help you also: cloud.google.com/compute/docs/instance-templates/… , instead of Image or Snapshot Commented Oct 6, 2021 at 15:17
  • I wrote an article on how to figure out startup times. This will also help you know where to look for startup script issues: jhanley.com/google-compute-startup-script-total-execution-time I am not sure which OS you are using, which is important to know which log files to look at. 2) Installing npm/node takes time, sometimes five minutes or more. When you say that you can access, define exactly what that means (the test and the error). Commented Oct 6, 2021 at 17:39
  • @JohnHanley I'll take a look at your article ! Basically when I do the setup manually I open my browser and do http://ipfrontend:3000 and I can access the frontend, but when I execute my script above, the instance are running (as u can see on the picture) but when I try to access http://ipfrontend:3000 I get the error in my browser because the frontend isn't running --> that's why I think my script-frontend isn't written correctly Commented Oct 6, 2021 at 18:26

1 Answer 1

Reset to default
1

Looking at the logs you included in the question, it looks like there could be a problem with your custom image. In line 7 of the log file, it warns that the directory “/mse-tsm-cloudsys-labl1/app/” is not found:

Oct  6 19:25:22 frontend-lab1 GCEMetadataScripts[795]: 2021/10/06 19:25:22 GCEMetadataScripts: startup-script: /tmp/metadata-scripts111777770/startup-script: line 3: cd: /mse-tsm-cloudsys-lab1/app/: No such file or directory

If this directory was used to contain your application files, it would be the cause of the subsequent errors related to NPM not being able to find files such as package.json and other files that belong to your application.

You should confirm whether the application path is correct or if the image was created correctly. You can follow this guide about creating custom images from your existing VM instances, and use the Node.js and Python guides to make sure you are following the correct steps.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.