Execute python script at launch time on Amazon Linux 2

Question

I am trying to execute a python script on an Amazon Linux 2 instance. In my user-data section I have a script which copies the python script from an S3 bucket to the instance and executes it like so:

#!/bin/bash

# e - stops the script if there is an error
# x - output every command in /var/log/syslog
set -e -x

# set AWS region
echo "export AWS_DEFAULT_REGION=us-east-1" >> /etc/profile
source /etc/profile
# copy python script from the s3 bucket
aws s3 cp s3://${bucket_name}/ /home/ec2-user --recursive

sudo python3 my_python_script.py

The problem is that the python script doesn't seem to be getting executed at all.

Note: the python script gets copied fine from the bucket

What I am missing here?

UPDATE: after checking /var/log/cloud-init-output.log it looks like the problem is in the python script, it cannot find the boto3 module:

+ python3 /home/ec2-user/my_python_script.py
Traceback (most recent call last):
  File "/home/ec2-user/my_python_script.py", line 1, in <module>
    import boto3
ModuleNotFoundError: No module named 'boto3'
Dec 10 15:52:25 cloud-init[3697]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Dec 10 15:52:25 cloud-init[3697]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Dec 10 15:52:25 cloud-init[3697]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed

The problem is that I do have boto3 module installed. I created a custom AMI image that does have all of the modules installed (I used pip3 to install them) before creating the custom AMI image

UPDATE2

I verified that the image does have boto3 package installed in the python3 library:

[ec2-user@ip-ip ~]$ python3
Python 3.7.9 (default, Aug 27 2020, 21:59:41)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>>

UPDATE3

The cause of the problem was that I installed the boto3 package for my user only (i.e. pip3 install boto3 --user) and then I created the AMI image. So after adding the bellow line to my user-data script it worked fine

#!/bin/bash

...
sudo pip3 install boto3
sudo python3 my_python_script.py

docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html "Scripts entered as user data are run as the root user, so do not use the sudo command in the script." — Tom Dalton
– Tom Dalton, Commented Dec 10, 2020 at 15:37
Hmm. you're copying the bucket content to /home/ec2-user but I'm not sure if the userdata executes from that directory - so python3 my_python_script.py maybe just can't find my_python_script.py in the working directory? — Tom Dalton
– Tom Dalton, Commented Dec 10, 2020 at 15:42
check /var/log/cloud-init-output.log to see whats going on. Also try executing the script manually from /var/lib/cloud/instances/<<instance-id>> — Asdfg
– Asdfg, Commented Dec 10, 2020 at 15:56
just do pip install boto3 before you execute the python script. — Asdfg
– Asdfg, Commented Dec 10, 2020 at 16:03

Chau Giang · Accepted Answer · 2020-12-10 15:56:48Z

1

You can redirect output to a file and read it to see the error: did you have python3, did your instance have credentianl/role to access this bucket, did you script requires any third party, can you try to run the script above as root in local first, the run command should be python3 /home/ec2-user/my_python_script.py?

edited Dec 10, 2020 at 15:56

answered Dec 10, 2020 at 15:35

Chau Giang

1,58415 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

14 Comments

Georgi Koemdzhiev Over a year ago

Thank you for your suggestion. However, I don't think that is the reason why my script is not getting executed. I have checked the /etc/profile file contents after the instance launches and it does contain my exported environment variable

Chau Giang Over a year ago

you should install it first @GeorgiKoemdzhiev, I am not familiar with python maybe pip install boto3 or pip3 install boto3 before sudo python3 /home/ec2-user/my_python_script.py should help

Chau Giang Over a year ago

I think you can specific path to python3, for example /usr/bin/python3 path/to/your/script (I got this when use which python3)

Chau Giang Over a year ago

You can try to add python3 to your PATH in /etc/environment also or use absolute python3 path instead of just python3 your/path/to/yourscript, just like other answer in the link above

Georgi Koemdzhiev Over a year ago

I will build a new image following your advice and see what happens :) Ideally, all of the packages should be available when I launch the server (i.e. I don't have to install them at boot time)

|

ashraf minhaj · Accepted Answer · 2022-11-27 18:19:03Z

0

For anyone who is using cloudinit and terraform, this is how I managed it to work (one part of multipart cloudinit user data)-

part {
    content_type      = "text/x-shellscript"
    filename          = "run_src.sh"
    content           = <<-EOF
      #!/bin/bash
      cd /root
      mkdir tmp
      mkdir tmp/origin
      mkdir tmp/converted
      mkdir tmp/packaged
      
      pip3 install boto3
      cd src

      python3 main.py

    EOF
  }

And it works like charm.

answered Nov 27, 2022 at 18:19

ashraf minhaj

1,2059 silver badges26 bronze badges

2 Comments

burglarhobbit Over a year ago

How did you include the main.py in the cloud init?

ashraf minhaj Over a year ago

See the part, you can actually include multiple parts. And you can also send files.

Collectives™ on Stack Overflow

Execute python script at launch time on Amazon Linux 2

2 Answers 2

14 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

14 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related