How to launch GCP Compute Engine VM with Startup Script via Python API?

Question

GCP has a published create_instance() code snippet available here, which I've seen on SO in a couple places e.g. here. However, as you can see in the first link, it's from 2015 ("Copyright 2015 Google Inc"), and Google has since published another code sample for launching a GCE instance dated 2022. It's available on github here, and this newer create_instance function is what's featured in GCP's python API documentation here.

However, I can't figure out how to pass a startup script via metadata to run on VM startup using the modern python function. I tried adding

    instance_client.metadata.items = {'key': 'startup-script',
                                      'value': job_script}

to the create.py function (again, available here along with supporting utility functions it calls) but it threw an error that the instance_client doesn't have that attribute.

GCP's documentation page for starting a GCE VM with a startup script is here, where unlike most other similar pages, it contains code snippets only for console, gcloud and (REST)API; not SDK code snippets for e.g. Python and Ruby that might show how to modify the python create_instance function above.

Is the best practice for launching a GCE VM with a startup script from a python process really to send a post request or just wrap the gcloud command

gcloud compute instances create VM_NAME \
  --image-project=debian-cloud \
  --image-family=debian-10 \
  --metadata-from-file=startup-script=FILE_PATH

...in a subprocess.run()? To be honest I wouldn't mind doing things that way since the code is so compact (the gcloud command at least, not the POST request way), but since GCP provides a create_instance python function I had assumed using/modifying-as-necessary that would be the best practice from within python...

Thanks!

Please don't use subprocess. Let's get the library|API working! — DazWilkin
– DazWilkin, Commented Jun 3, 2022 at 0:28
this aligns with my hunch on what the right solution should be, but I gotta admit for me it's really more of a hunch than anything principled. can you elaborate on the downside of relying too much on subprocess for something like this? certainly people use bash scripts in production processes successfully... — Max Power
– Max Power, Commented Jun 3, 2022 at 0:37
Subprocess invokes a (bash) subprocess and runs the desired process. The "interface" is stdin, stdout and stderr which are a bunch of streams of string. When you invoke a Python method directly, you get all the power of the language's runtime mechanisms such as typed (ok, it's Python...) inputs and outputs, the ability to try the method and the ability to capture structured errors. I think Subprocess should only be used when you can't use native language bindings and when you specifically want to check a bash invocation of a binary. — DazWilkin
– DazWilkin, Commented Jun 3, 2022 at 1:34
one last little quibble here (though maybe I'm wrong) on your characterization of subprocess having an interface of only "stdin, stdout and sterr", and being "string-based streams all the way down." The return I get from subprocess includes an exit-code, not just those streams. So I can still do some structured exception handling for non-zero exit codes without log-processing, though I agree we can do much more fine-grained exception handling via non-subprocess library/sdk. — Max Power
– Max Power, Commented Jun 3, 2022 at 3:04
Right, the shell's exit code but you're getting probably a 1 whereas you could be getting Errors. My point is that using subprocess is very lossy. Why lose info if you don't have to? — DazWilkin
– DazWilkin, Commented Jun 3, 2022 at 3:10

DazWilkin · Accepted Answer · 2022-06-03 03:11:25Z

3

So, the simplest (!) way with the Python library to create the equivalent of --metadata-from-file=startup-scripts=${FILE_PATH} is probably:

from google.cloud import compute_v1

instance = compute_v1.Instance()

metadata = compute_v1.Metadata()
metadata.items = [
    {
        "key":"startup-script",
        "value":'#!/usr/bin/env bash\necho "Hello Freddie"'
    }
]

instance.metadata = metadata

And another way is:

metadata = compute_v1.Metadata()

items = compute_v1.types.Items()
items.key = "startup-script"
items.value = """
#!/usr/bin/env bash

echo "Hello Freddie"
"""

metadata.items = [items]

NOTE In the examples, I'm embedding the content of the FILE_PATH in the script for convenience but you could, of course, use Python's open to achieve a more comparable result.

It is generally always better to use a library|SDK if you have one to invoke functionality rather than use subprocess to invoke the binary. As mentioned in the comments, the primary reason is that language-specific calls give you typing (more in typed languages), controlled execution (e.g. try) and error handling. When you invoke a subprocess its string-based streams all the way down.

I agree that the Python library for Compute Engine using classes feels cumbersome but, when you're writing a script, the focus could be on the long-term benefits of more explicit definitions vs. the short-term pain of the expressiveness. If you just wanna insert a VM, by all means using gcloud compute instances create (I do this all the time in Bash) but, if you want to use a more elegant language like Python, then I encourage you to use Python entirely.

CURIOSITY gcloud is written in Python. If you use Python subprocess to invoke gcloud commands, you're using Python to invoke a shell that runs Python to make a REST call ;-)

edited Jun 3, 2022 at 3:11

answered Jun 3, 2022 at 1:45

DazWilkin

41.7k8 gold badges61 silver badges113 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Max Power Over a year ago

hey thanks this is awesome. just verifying I can get it to work and will shortly accept. I gotta ask though - are the valid keys we can pass to metadata items documented anywhere? I tried some more googling and found this page which seemed like it should help but didn't. developers.google.com/resources/api-libraries/documentation/… ...and I tried inspecting the Metadata() object before assigning anything to it but it didn't have any set of keys yet...

DazWilkin Over a year ago

Have a look at e.g. VM Metadata. There are 2 types: project-level and instance-level. You can also use gcloud instances describe. Metadata is managed by the Metadata service. If you create an instance with startup-script as above, from a shell on the instance, you can then

curl --header "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/startup-script"

to read the value (the script).

Max Power Over a year ago

hm, I still couldn't find on the page you linked, or 2-3 links I followed from it, where it actually specifically documents this behavior. But regardless, no worries, super appreciate your answer (which I've confirmed works and accepted now) however you came to learn it. I actually wouldn't even say the python library is cumbersome, given that I could download the main functions from the docs easily and they're quite convenient to call as-is. And re python -> gcloud -> python...don't tempt me with a good time!

DazWilkin Over a year ago

Here's a summary of default values.

DazWilkin Over a year ago

I worked at Google in Google Cloud Platform and I relish questions like yours :-)

|

richmake · Accepted Answer · 2024-02-10 03:40:19Z

Had to make a few changes, and use the same fingerprint:

            import googleapiclient.discovery
            from google.oauth2 import service_account
            compute = googleapiclient.discovery.build('compute', 'v1', credentials=GCPClass.credentials)
            metadata_response = compute.instances().get(project=GCPClass.project, zone=zone, instance=vmname).execute()
            current_fingerprint = metadata_response['metadata']['fingerprint']
            compute_client = compute_v1.InstancesClient(credentials=GCPClass.credentials)
            metadata = compute_v1.Metadata()
            items = compute_v1.types.Items()
            items.key = "startup-script"
            items.value = script_content_crlf
            metadata.fingerprint = current_fingerprint
            metadata.items = [items]
            request = {
                "project": GCPClass.project,
                "zone": zone,
                "instance": vmname,
                "metadata_resource": metadata
            }
            response = compute_client.set_metadata(**request)
            print(response.status, response.result(), response.done())

Collectives™ on Stack Overflow

How to launch GCP Compute Engine VM with Startup Script via Python API?

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related