Bash scripting, checking for errors, logging

Question

Here's one for the bash-fu wizards. No, actually, I'm just kidding, you'll all probably know this except for me..

I'm trying to create a backup shell script. The idea is fairly simple: find files in a certain folder, older than 7 days, tar/gzip them to another directory, and remove them. The problem is, I'm not sure if I'll have enough permissions to create a tar/gzip file in the target dir. Is there any (proper) way to check if the file has been created successfully, and if so, delete the files. Otherwise, skip that part and don't destroy customers' data. I hear they are not very fond of that.

Here's what I have so far:

01: #!/bin/bash
02: 
03: ROOTDIR="/data/www"
04: 
05: TAR="${ROOTDIR}/log/svg_out_xml/export_out_ack_$(date +%Y-%m-%d).tar"
06: cd ${ROOTDIR}/exchange/export/out_ack/
07: find . -mtime +7 -type f -print0 | xargs -0 tar -cf "${TAR}"
08: gzip ${TAR}
09: find . -mtime +7 -type f -print0 | xargs -0 rm -f

Basically, I'd need to check if everything went fine on lines 7 and 8, and if so execute 9.

Additionally, I'd like to make a log file of these operations so I know everything went fine (this is a nightly cron job).

Here come all the comments, use something pre-made its safe. — Recursion
– Recursion, Commented Mar 2, 2010 at 10:05
I have accepted Dennis Williamson's answer because he was the only one to address the logging issue, but all suggestions have been quite helpful, thanks guys! — dr Hannibal Lecter
– dr Hannibal Lecter, Commented Mar 2, 2010 at 16:13

Idelic · Accepted Answer · 2010-03-02 16:43:53Z

For logging, you can arrange for all output written on standard output and/or standard error to go to a file. That way, you don't need to redirect the output of each command:

# Save standard output and standard error
exec 3>&1 4>&2
# Redirect standard output to a log file
exec 1>/tmp/stdout.log
# Redirect standard error to a log file
exec 2>/tmp/stderr.log

# Now the output of all commands goes to the log files
echo "This goes to /tmp/stdout.log"
echo "This goes to /tmp/stderr.log" 1>&2
...

# Print a message to the original standard output (e.g. terminal)
echo "This goes to the original stdout" 1>&3

# Restore original stdout/stderr
exec 1>&3 2>&4
# Close the unused descriptors
exec 3>&- 4>&-

# Now the output of all commands goes to the original standard output & error
...

To execute a command only if a previous one succeeds, you can chain them with conditionals:

# Execute command2 only if command1 succeeds, and command3 only if both succeed:
command1 && command2 && command3

# Execute command2 only if command1 fails
command1 || command2

so you can do things like

{ find . -mtime +7 -type f -print0 | xargs -0 tar -cf "${TAR}" &&
  gzip ${TAR} && 
  find . -mtime +7 -type f -print0 | xargs -0 rm -f } || 
    { echo "Something failed" 1>&2; exit 1 }

or provide details in the log output:

find . -mtime +7 -type f -print0 | xargs -0 tar -cf "${TAR}" || 
  { echo "find failed!!" 1>&2; exit 1 }
gzip ${TAR} || 
  { echo "gzip failed!!" 1>&2; exit 1 }
find . -mtime +7 -type f -print0 | xargs -0 rm -f || 
  { echo "cleanup failed!!" 1>&2; exit 1}

Dennis Williamson · Accepted Answer · 2010-03-02 13:59:28Z

8

For logging, you can wrap sections of your script in curly braces and redirect the stdout to a log file:

{
    script_command_1
    script_command_2
    script_command_3
} >> /path/to/log_file

answered Mar 2, 2010 at 13:59

Dennis Williamson

364k95 gold badges386 silver badges446 bronze badges

1 Comment

dr Hannibal Lecter Over a year ago

Kewl, I hope it will work if I write it like this: } >> ${LOG} 2>&1

falstro · Accepted Answer · 2010-03-02 10:06:32Z

6

The easy way out, but with no explicit error message add -e to the shebang, i.e. #!/bin/sh -e which will cause the shell to exit if a command fails..

Cron should give you an error message through the mail I guess though.

If you want to go full blown back-up scheme though, I'd suggest you use something that has already been made. There are a bunch out there, and most work very well.

answered Mar 2, 2010 at 10:06

falstro

35.9k11 gold badges76 silver badges88 bronze badges

4 Comments

Martin Wickman Over a year ago

Excellent suggestion. I always use -e in my scripts.

dr Hannibal Lecter Over a year ago

Thanks for the tip, very useful! Unfortunately, cron won't send us anything as we are not the administrators of this system. It would be nice though..

falstro Over a year ago

@dr: It will if you define the MAILTO in your crontab.

Dennis Williamson Over a year ago

See BashFAQ/105 regarding bash -e. tl;dr - "don't use set -e"

Ignacio Vazquez-Abrams · Accepted Answer · 2010-03-02 10:07:52Z

3

GNU tar has --remove-files that will remove files once they've been added to the archive. v will cause it to list out files as it adds them. z will pipe the tar through gzip on the fly.

Your find solution is racy; a file may fit the criteria in between invocations, thus getting deleted but not backed up.

answered Mar 2, 2010 at 10:07

Ignacio Vazquez-Abrams

804k160 gold badges1.4k silver badges1.4k bronze badges

1 Comment

dr Hannibal Lecter Over a year ago

Thanks for your answer. It is highly unlikely that a file will fit between invocations, since this script will run at 4AM when no files should be created in the folder I'm backing up. Of course, there's always that one user..

Collectives™ on Stack Overflow

Bash scripting, checking for errors, logging

4 Answers 4

Comments

1 Comment

4 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

1 Comment

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related