Essential tips for scripting in Bash and Python

In my last article, Developers want scripts and tools — not “how-to” guides, I wrote about the importance of providing runnable script files in your software-related technical writing, instead of just listing out a series of commands that the reader has to copy and paste in order to accomplish a task.

In this article, I’ll walk you through some tips that have been indispensable to me while writing Bash and Python scripts, and distributing those scripts to other engineers.

Note: this guide is targeted towards Linux users. In future articles, I’ll be writing about PowerShell scripting in Windows (and why it’s awesome).

Bash scripts

For simple scripts that only consist of a few commands and maybe a few optional inputs, I like to use Bash, since it requires very little boilerplate and the syntax is extremely minimal, requiring few keystrokes and therefore saving lots of time.

• To create a new Bash script, you can use the new-script tool that I mentioned in my previous article. After installing it, you can run new-script my-cool-utility to create an executable script named my-cool-utility with useful boilerplate already inserted, and immediately launch into your preferred editor to start authoring the script. Zero-friction! To install this tool, just run this one command:

sudo bash -c 'cd /usr/local/bin && wget -qO-    https://gist.githubusercontent.com/bduffany/a0cb3a8be0364dae91804ac510670e81/raw/7717ce1911fff3ba19dabc7f4acbb03cd5beda1f/new-script.sh > new-script && chmod +x new-script'

• If your script relies on packages installed using apt-get, just install those packages directly in your script instead of asking users to install it before running the script! It’s usually fine to just sudo apt install -y. A good recipe for detecting whether something is installed is to just run type $COMMAND || sudo apt install -y $PACKAGE.
• If your script needs to update a file that the user might already have (such as .bashrc), make sure to use grep to check that the user hasn’t made that same modification to their file already. Otherwise, you may wind up writing duplicate data or messing with some setting that the user already wrote themselves.
• Gather information from the user using read prompts, e.g. read -P "Enter your API key: " api_key. For more sensitive info like passwords, tack on the -s flag, which prevents the user’s input from being displayed on the console.
• Make use of the built-in environment variables. Instead of asking the user for their username, just reference $USER. $HOME is also very useful.
• To find and replace text in files or input text, my go to recipe isperl -p -i -e 's@foo@foo_replacement@' file_name.txt. You’ll often see s/foo/foo_replacement/ instead, but I prefer to use @ instead of / as the delimiter since I often find it useful to manipulate directory name patterns, which already contain /.
• Instead of telling your readers to add things to their ~/.bashrc file, you can do it for them! For example, this simple script checks whether a particular alias is already defined in ~/.bashrc, and if not, it adds the alias:

• The mktemp command is a great way to quickly create temporary files or temporary working directories in the /tmp/directory, which will automatically be deleted on the next system reboot.
• In some cases, your script might need to create some temporary files outside of /tmp/ or make temporary modifications to files. In those cases, use cleanup traps to undo any damage or clean up any garbage generated by your script.
• Make sure to abort the script if any commands fail, using set -eo pipefail at the top of the script (just after the hashbang). (new-script does this for you!)
• Use the rsync command to efficiently synchronize one directory’s contents with another (contents of the source or target directory can come from a server!). This is especially useful if your script needs to mirror some data to the user’s local filesystem, or create backups of data in a pre-determined directory.

If your Bash script starts getting too complicated (you find yourself needing to use loops, arrays, maps, or command line argument parsing), it’s probably time to switch to Python!

Python scripts

I use Python scripts whenever I need to parse arguments (which is a terrible experience in Bash) or if I need to use complex data structures (the syntax for dealing with maps and arrays in Bash is atrocious!).

This article is not intended to be a complete guide to Python, but here are a few basic tips for writing command-line tools in Python:

• For starters, some things are much easier to do in Bash than in Python. Inevitably, you will want to intermix some Bash commands into your Python script to save time. For that, I always keep this nifty utility function in my toolbox:

• If your Python script is expecting to be run inside a particular directory, make sure that the script automatically changes to that directory! The recipe os.chdir(os.path.dirname(path)) is your friend if you want to change to the same directory as the file passed in as the input argument.
• Use the argparse module to deal with command line arguments. The official documentation has some fantastic examples.
• The tempfile module is your friend if you need to create temporary files that will get automatically cleaned up by the operating system. The official documentation includes some great examples.
• Use os.walk to recursively process directories and their children.
• If your script needs dependencies that aren’t included in the Python standard library, you can include a wrapper script in Bash which automatically installs them, like the one below, which uses Pex to run the Python script in an isolated virtual environment with your required dependencies:

• Use joblib.Memory to cache the results of expensive operations. For example, if your script runs an expensive query against your database and you don’t want users to rack up your compute costs, you can save the returned result to disk for subsequent invocations.

Hopefully you found these tips to be useful. If you have any other tips, I’d love to hear them. Thanks for reading!