SSH capture exit code of remote ssh commands

Question

I have written a script, that is part of a gitlab ci pipeline, to deploy code via ssh to a remote server. The script is located locally and forwared to the remote server. That works okay so far, but unfortunately there is a problem when errors occur. Right now the migrate command exits, because there are no db credentials saved. The command exits with a non zero code, but the pipeline still succeeds, which is misleading, and forces me to login to the remote server and check the logs there.

How can I forward the exit codes from the remote to the local machine, so that in this case the gitlab-ci job also fails?

I've come across these Q/A's, but since I'm fairly new to bash I don't understand the solution. Removing set -e didn't help.

The gitlab-ci config part looks like this:

script:
# Change directory to the packaged app (not the repo).
- cd /var/www/html

# Copy over the code.
- |
  rsync \
    -azc \
    --exclude-from="$RSYNC_EXCLUDES" \
    "$(pwd)/" "$SSH_CONNECTION:$BASE_DIR/releases/$CI_COMMIT_SHA/"

# Run the post deployment script.
- ssh -T "$SSH_CONNECTION" "bash -s" < ./.deploy/post_deploy.sh "$BASE_DIR" "$CI_COMMIT_SHA"

The post_deploy.sh script looks like this:

#!/bin/bash

# Terminate execution if any command fails
set -e

########################
# Program arguments:
########################

## The base directory
BASE_DIR=$1

# The commit sha that is being deployed.
COMMIT_SHA=$2

################
# Variables:
################

# The path to the releases directory
RELEASE_DIR=$BASE_DIR/releases/$COMMIT_SHA

# The path to the shared directory.
SHARED_DIR=$BASE_DIR/shared

###############
# Deployment:
###############

echo "Symlink $SHARED_DIR/.env to $RELEASE_DIR/.env."
rm -rf "$RELEASE_DIR/.env" && ln -sf "$SHARED_DIR/.env" "$RELEASE_DIR/.env."

echo "Symlink $SHARED_DIR/storage to $RELEASE_DIR."
rm -rf "$RELEASE_DIR/storage" && ln -sf "$SHARED_DIR/storage" "$RELEASE_DIR"

echo "Fixing permissions."
find "$RELEASE_DIR" -type f -exec chmod 644 {} \;
find "$RELEASE_DIR" -type d -exec chmod 755 {} \;

echo "Running custom scripts."
php "$RELEASE_DIR" artisan storage:link
php "$RELEASE_DIR" artisan migrate --no-interaction --force

php "$RELEASE_DIR" artisan cache:clear
php "$RELEASE_DIR" artisan config:clear

php "$RELEASE_DIR" artisan config:cache
php "$RELEASE_DIR" artisan view:cache

echo "Releasing the new version (symlink current/)."
ln -nsf "$RELEASE_DIR/" "$BASE_DIR/current"

echo "Remove all releases older than a day."
find "$BASE_DIR/releases" -maxdepth 1 -mindepth 1 -mtime +1 -exec rm -r {} \;

Novikov · Accepted Answer · 2020-08-26 21:42:43Z

2

it should work if you really get non zero code. I tested on my machines

$ ssh user@host 'bash -s' < script.sh 
remote script execute with exit 1
$ echo $?
1

Gitlab must fail a job if at least one script line returns non zero.

As far as I understand you except you said about this line

php "$RELEASE_DIR" artisan migrate --no-interaction --force

I don't know about this tool, but often flags --force should make command without zero code. Please check the flags.

If that doesn't work, try putting

if [[ $? != 0 ]]; then
  exit 1
fi

after line with migrate

answered Aug 26, 2020 at 21:42

Novikov

7607 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

mbomb007 Over a year ago

Will it still work if instead of a separate script file, you use something like ssh user@host /bin/bash <<EOF [commands] EOF || exit $?, with the commands on multiple lines in the gitlab-ci.yml file?

drewjosh Over a year ago

This worked great! Many thanks!

Tom · Accepted Answer · 2020-08-27 08:10:34Z

1

Okay. It seems that this was a mistake on my end, and everything is working correctly. I wasn't calling one of the commands correctly:

php "$RELEASE_DIR" artisan migrate --no-interaction --force

This fixes it:

php "$RELEASE_DIR/artisan" migrate --no-interaction --force

answered Aug 27, 2020 at 8:10

Tom

4,7825 gold badges26 silver badges53 bronze badges

Collectives™ on Stack Overflow

SSH capture exit code of remote ssh commands

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related