“Foolproof Your Bash Script” – Some Best Practices

I am a DevOps practitioner and a lazy one too, so whenever we come across any task that needs to be repeated, we create a bash script. I have been doing this for a long time and after doing a lot of mistakes, I figured that if we follow some basic rules we can make our script more portable and less prone to failure when semantics change. These rules and practices have been discussed further in the blog.

1. Always start with a shebang

The first rule of shell scripting is that you always start your scripts with a shebang also called sha-bang, hashbang, pound-bang, or hash-pling. While the name might sound funny the shebang line is very important; it tells the system which binary to use as the interpreter for the script. Without the shebang line, the system doesn’t know what language to use to process the script.

[js]#!/bin/bash[/js]

A typical bash shebang line would look like the following:

2. Use Built-in Shell Options

Use set -o errexit (a.k.a. set -e) to make your script exit when a command fails.  add || true to commands that you allow to fail.
Use set -o nounset (a.k.a. set -u) to exit when your script tries to use undeclared variables.
Use set -o xtrace (a.k.a set -x) to trace what gets executed. Useful for debugging (optional).
Use set -o pipefail in scripts to catch mysqldump fails in e.g. mysqldump |gzip. The exit status of the last command that threw a non-zero exit code is returned.

[js]#!/bin/bash
set -o nounset
set -o errexit[/js]

3. Naming conventions for Variables

1. Strings should have the form name=value.
   Ideally variable names should only consist uppercase letters, digits, and the ‘_’ (underscore).
2. Variable Names shall not contain the character ‘=’ or ‘-‘
   $ running-process-id=`ps -eopid`\
3. No spaces before or after that equal sign
   $ language = PHP
   -bash: language: command not found
4. Double quotes around every parameter expansion
   $ song="My song.mp3"
$ rm $song
   

   [js]rm: My: No such file or directory
   rm: song.mp3: No such file or directory [/js]

   $ rm "$song"

5. Don’t start Variable Name with special characters or Numbers
   $ -song="My song.mp3"
   

   [js]-song=my song.mp3: command not found [/js]

   $ 123song="My song.mp3"
   

   [js]123song=my song.mp3: command not found[/js]

4. Variable Annotations

Bash allows for a limited form of variable annotations. The most important ones are:
local (for local variables inside a function)
readonly (for read-only variables)

Strive to annotate almost all variables in a bash script with either local or readonly.

5. Use $() over backticks

Avoid using backticks ““”, they are hard to read and in some fonts easily confused with single quotes. A lot of quoting needed in nesting.

Example:

$ echo "one-$(echo two-$(echo three-$(echo four)))"

[js]one-two-three-four[/js]

$ echo "one-`echo two-\`echo three-\`\`echo four\`\`\``"

[js]one-two-three-four [/js]

6. Prefer using Double Brackets

Let me illustrate how [[ can be used and how it can help you to avoid some of the common mistakes made by using [] or test:

$ var=''
$ [ $var = ''" ] && echo True

[js]-bash: [: =: unary operator expected[/js]

$ [ "$var" = '' ] && echo True

[js]True[/js]

$ [[ $var = '' ]] && echo True

[js]True[/js]

Using quotes, [ “$var” = ” ] expands into [ “” = ” ] and test has no problem.

Now, [[ can see the whole command before it’s being expanded. It sees $var, and not the expansion of $var. As a result, there is no need for the quotes at all! [[ is safer.

$ var=
$ [ "$var" < a ] && echo True

[js]-bash: a: No such file or directory[/js]

$ [ "$var" \< a ] && echo True


[js]True[/js]

$ [[ $var < a ]] && echo True


[js]True[/js]

In this example, we attempted a string comparison between an empty variable and ‘a’. We’re surprised to see the first attempt does not yield True even though we think it should. Instead, we get some weird error that implies Bash is trying to open a file called ‘a’.

We’ve been bitten by File Redirection. Since test is just an application, the < character in our command is interpreted (as it should) as a File Redirection operator instead of the string comparison operator of test. Bash is instructed to open a file ‘a’ and connect it to stdin for reading. To prevent this, we need to escape < so that test receives the operator rather than Bash. This makes our second attempt work.

Using [[ we can avoid the mess altogether. [[ sees the < operator before Bash gets to use it for Redirection — problem fixed. Once again, [[ is safer. Even more dangerous is using the > operator instead of our previous example with the < operator. Since > triggers output Redirection it will create a file called ‘a’. As a result, there will be no error message warning us that we’ve committed a sin! Instead, our script will just break. Even worse, we might overwrite some important file! It’s up to us to guess where the problem is:

$ var=a
$ [ "$var" > b ] && echo True || echo False

[js]True[/js]

$ [[ "$var" > b ]] && echo True || echo False


[js]False[/js]

Two different results, not good. The lesson is to trust [[ more than [. [ “$var” > b ] is expanded into [ “a” ] and the output of that is being redirected into a new file called ‘b’. Since [ “a” ] is the same as [ -n “a” ] and that basically tests whether the “a” string is non-empty, the result is a success and the echo True is executed.

Using [[ we get our expected scenario where “a” is tested against “b” and since we all know “a” sorts before “b” this triggers the echo False statement. And this is how you can break your script without realizing it. You will however have a suspiciously empty file called ‘b’ in your current directory.

Yes it adds a few characters, but [[ is far safer than [. Everybody inevitably makes programming errors. Even if you try to use [ safely and carefully avoid these mistakes, I can assure you that you will make them. And if other people are reading your code, you can be sure that they’ll absolutely mess things up.

7. Regular Expressions/Globbing

[[ provides the following features over [ :-

t=”abc123″
[[ “$t” == abc* ]] # true (globbing)
[[ “$t” == “abc*” ]] # false (literal matching)
[[ “$t” =~ [abc]+[123]+ ]] # true (regular expression)
[[ “$t” =~ “abc*” ]] # false (literal matching)
Note, that starting with bash version 3.2 the regular or globbing expression
must not be quoted. If your expression contains whitespace you can store it in a variable:
r=”a b+”
[[ “a bbb” =~ $r ]] # true

Globbing based string matching is also available via the case statement:

[js]case $t in
abc*) ;;
esac[/js]

8. Avoiding Temporary Files

Some commands expect filenames as parameters so straightforward pipelining does not work.
This is where <() operator comes in handy as it takes a command and transforms it into something
which can be used as a filename:

diff <(wget -O - url1) <(wget -O - url2)

Also useful are “here documents” which allow arbitrary multi-line string to be passed
in on stdin. The two occurrences of ‘EOF’ brackets the document.

# DELIMITER is an arbitrary string

[js]command << EOF
Some importent text
$(cmd)
EOF[/js]

‘EOF’ can be any text.

9. Debugging

To perform a syntax check/dry run of your bash script run:

bash -n myscript.sh

To produce a trace of every command executed run:

bash -v myscripts.sh

To produce a trace of the expanded command use:

bash -x myscript.sh

-v and -x can also be made permanent by adding
set -o verbose and set -o xtrace to the script prolog.
This might be useful if the script is run on a remote machine, e.g.
a build-bot and you are logging the output for remote inspection.

10. Trap forced exit of script

Don’t let your script exit unexpectedly, trap when someone update press ctrl+c and exit from your script gracefully.

[js]# trap ctrl-c and call ctrl_c()
trap ctrl_c INT

function ctrl_c() {
echo "** Trapped CTRL-C"
}

for i in `seq 1 5`; do
sleep 1
echo -n "."
done[/js]

This is how simple it is to Foolproof  your bash scripts. In my next blog, I will talk about more interesting features available in Linux OS.