Bash on dwmkerr.com

A simple Makefile 'help' command

Sun, 25 Sep 2022 00:00:00 +0000

In this article I’m going to show you how to add a make help command to your makefiles that quickly and easily shows simple documentation for your commands:

To add the help command to your makefile, add a recipe like so:

.PHONY: help
help: # Show help for each of the Makefile recipes.
 @grep -E '^[a-zA-Z0-9 -]+:.*#' Makefile | sort | while read -r l; do printf "\033[1;32m$$(echo $$l | cut -f 1 -d':')\033[00m:$$(echo $$l | cut -f 2- -d'#')\n"; done

Now just make sure that each of your recipes has a comment that follows the recipe name, which will be used as its documentation. For example, my website repository has the following recipes in the makefile:

default: help

.PHONY: help
help: # Show help for each of the Makefile recipes.
 @grep -E '^[a-zA-Z0-9 -]+:.*#' Makefile | sort | while read -r l; do printf "\033[1;32m$$(echo $$l | cut -f 1 -d':')\033[00m:$$(echo $$l | cut -f 2- -d'#')\n"; done

.PHONY: setup
setup: # Setup tools required for local development.
 brew install hugo
 hugo version
 git submodule update --init --recursive --remote

.PHONY: newpost
newpost: # Create a new post.
 cd dwmkerr.com; hugo new posts/my-first-post.md

.PHONY: serve
serve: # Serve the site locally for testing.
 cd dwmkerr.com; hugo server --baseURL "http://localhost/" --buildDrafts -v --debug

.PHONY: build # Build the site.
build:
 cd dwmkerr.com; hugo --minify

With this setup, you can just enter make, or make help, to see the output below:

$ make help
help: Show help for each of the Makefile recipes.
newpost: Create a new post.
serve: Serve the site locally for testing.
setup: Setup tools required for local development.

Simple! You can find the code at:

https://github.com/dwmkerr/makefile-help

How it Works

This project was inspired by the project golang-cli-template, which I noticed had this cool feature of showing help for the makefile commands.

I built my own version of the command, which is a little bit more verbose, but I think a little easier to read and parse. I’ve also included the original version, with a link to the source in the makefile-help repo.

Essentially, the code simply:

Searches for recipes - these are lines that start with text, have a colon and a hash symbol
Goes through each recipe found, extracts the recipe name and documentation comment
Write each of the recipe names and its documentation to the console

Testing Scripts

I wanted to make sure that if I improve on the script over time, or add different versions, the code won’t break. There’s a test script, which is a simple shell script that runs the two help commands and assets the output is as expected.

At the time of writing, the shell script looks like this:

#!/usr/bin/env bash

set -e

recipes=("help" "help-compact")

# Some colour codes for formatting.
green="\033[1;32m"
red="\033[1;31m"
reset="\033[00m"

# Default to success for the result of tests.
result=0

for recipe in "${recipes[@]}"; do
 # Create the path to the expected output file.
 expected_output="./test-cases/${recipe}-expected-output.txt"
 if [ ! -f "${expected_output}" ]; then
 printf "[${red}FAIL${reset}] '${recipe}' failed, test file '${expected_output}' not found\n"
 result=1
 elif [ "$(make ${recipe})" == "$(cat ${expected_output})" ]; then
 printf "[${green}PASS${reset}] '${recipe}' passed\n"
 else
 printf "[${red}FAIL${reset}] '${recipe}' failed\n"
 result=1
 fi
done

# Return the exit code.
exit ${result}

One thing that is nice about the tests is that they are incorporated into a GitHub Action, which runs the tests using Ubuntu, MacOS and Windows, and tests on both Bash and the generic sh shell.

This uses the following features of GitHub actions:

Runner Images - predefined images are made available by GitHub for various operating systems
Shell Specificity - GitHub actions allow you to specify the shell used for a step
GitHub Action Matrix Strategies - A matrix of operating systems is specified, to avoid duplicating the pipeline steps for each supported operating system

This project provides a nice template or starting point if you want to build a simple shell script with some basic testing features.

Effective Shell for Beginners

Tue, 21 Jan 2020 00:00:00 +0000

I have rebuilt my “Effective Shell” series as an online book - it’s available now on:

https://effective-shell.com

The whole site is built from a GitHub repo at github.com/dwmkerr/effective-shell. It is open for contributions, changes, issues and suggestions. I’ve also added a comment section to each page to get input.

To keep the material as accessible as possible, I have added a new section for beginners, to help anyone who has not used a shell before. It goes over who the book is useful for, what the shell is, and how to set up your computer to work through the material:

All comments and suggestions are welcome!

Effective Shell Part 7: The Subtleties of Shell Commands

Tue, 25 Jun 2019 07:25:23 +0000

In this chapter, we’ll take a look at the various different types of shell commands that exist and how this can affect your work.

By the end of this chapter, you might even be able to make sense of the horrifying and perfectly syntactically valid code below:

which $(where $(what $(whence $(whereis who))))

What Are Commands?

This is really important to understand! A command in a shell is something you execute. It might take parameters. Generally it’ll have a form like this:

command param1 param2

We’ve already seen many commands during this series:

ls # Show the contents of the current directory
cd ~ # Move to the user's home
cat file.txt # Output the contents of 'file.txt' to stdout

But to be an effective shell user, you must understand that not all commands are created equal. The differences between the types of commands will affect how you use them.

There are four types of commands in most shells:

Executables
“Built-Ins” (which we’ll just call builtins from now on)
Functions
Aliases

Let’s quickly dig in and see a bit more.

Executables - Programs

Executables are just files with the ’executable’ bit set¹. If I execute the cat command, the shell will search for an executable named cat in my $PATH. If it finds it, it will run the program.

$ cat file.txt
This is a simple text file

What is $PATH? $PATH is the standard environment variable used to define where the shell should search for programs. If we temporarily empty this variable, the shell won’t find the command:

$ PATH="" cat file.txt
bash: cat: No such file or directory

Normally your $PATH variable will include the standard locations for Linux programs - folders such as /bin, /sbin, /usr/bin and so on².

If you were to print the variable, you’d see a bunch of paths (they are separated by colons; I’ve put them on separate lines for readability):

/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin

The shell will start with the earlier locations and move to the later ones. This allows local flavours of tools to be installed for users, which will take precedence over general versions of tools.

There will likely be other locations too - you might see Java folders, package manager folders and so on.

Executables - Scripts

Imagine we create a text file called dog in the local folder:

#!/bin/sh
echo "🐶 woof 🐶"

If we make the file executable, by running chmod +x dog³, then we can run this just like any other program - as long as we tell the shell to look for programs in the current directory:

$ PATH="." dog
🐶 woof 🐶

More common would be to run the program by giving a path:

$ ./dog
🐶 woof 🐶

Or just move it to a standard location that the shell already checks for programs:

$ mv dog /usr/local/bin
$ dog
🐶 woof 🐶

The point is that executables don’t have to be compiled program code. If a file starts with #! (the ‘shebang’), then the system will try to run the contents of the file with the program specified in the shebang.

We will look at shebangs in greater detail in a later chapter.

Builtins

OK, so we’ve seen executables. What about a command like this?

local V="hello" echo $V

You will not find the local executable anywhere on your system. It is a builtin - a special command built directly into the shell program.

Builtins are often highly specific to your shell. They might be used for programming (local for example is used to declare a locally scoped variable), or they might be for very shell-specific features.

This is where we need to take note. As soon as you are running a builtin, you are potentially using a feature that is specific to your shell, rather than a program that is shared across the system and can be run by any shell.

Trying to programmatically execute local as a process will fail - there is no executable with that name; it is purely a shell construct.

So how do we know if a command is a builtin? The preferred method is to use the type command:

$ type local
local is a shell builtin

The type command (which is itself a builtin!) can tell you the exact type of shell command.

Interestingly, you might be using more builtins than you think. echo is a program, but most of the time you are not executing it when you are in a shell:

$ type -a echo
echo is a shell builtin
echo is /bin/echo

By using the -a flag on type to show all commands that match the name, we see that echo is actually both a builtin and a program.

Many simple programs have builtin versions. The shell can execute them much faster.

Some commands are a builtin so that they can function in a sensible manner. The cd command changes the current directory - if we executed it as a process, it would change only the directory for the cd process itself, not the shell, making it much less useful.

Builtins will vary from shell to shell, but many shells are ‘Bash-like’ - meaning they will have a set very similar to the Bash builtins, which you can see here:

https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html

As should be familiar from Part 3: Getting Help, you can get help for builtins:

$ man source # source is a builtin
BUILTIN(1) BSD General Commands Manual BUILTIN(1)

NAME
 builtin, !, %, # ...snip...

SYNOPSIS
 builtin [-options] [args ...]

However, the manual will not show information on specific builtins, which is a pain. Your shell might have an option to show more details - for example, in Bash you can use help:

$ help source
source: source filename [arguments]
 Read and execute commands from FILENAME and return. The pathnames
 in $PATH are used to find the directory containing FILENAME. If any
 ARGUMENTS are supplied, they become the positional parameters when
 FILENAME is executed.

But remember: help is a builtin; you might not find it in all shells (you won’t find it in zsh, for example). This highlights again the challenges of builtins.

Functions

You can define your own shell functions. We will see a lot more of this later, but let’s show a quick example for now:

$ restart-shell () { exec -l "$SHELL" }

This snippet creates a function that restarts the shell (quite useful if you are messing with shell configuration files or think you might have irreversibly goofed up your current session).

We can execute this function just like any command:

$ restart-shell

And running type will show us that this is a function:

$ type restart-shell
restart-shell is a function
restart-shell ()
{
 exec -l $SHELL
}

Functions are one of the most powerful shell constructs we will see; they are extremely useful for building sophisticated logic. We’re going to see them in a lot more detail later, but for now it is enough to know that they exist, and can run logic, and are run as commands.

Aliases

An alias is just a shortcut. Type in a certain set of characters, and the shell will replace them with the value defined in the alias.

Some common commands are actually already aliases - for example, in my zsh shell, the ls command is an alias:

% type -a ls
ls is an alias for ls -G
ls is /bin/ls

I make sure that when I use the ls command, the shell always expands it to ls -G, which colours the output.

We can quickly define aliases to save on keystrokes. For example:

$ alias k='kubectl'

From this point on, I can use the k alias as shorthand for the kubectl command.

Aliases are far less sophisticated than functions. Think of them as keystroke savers and nothing more, and you won’t go far wrong. Aliases are not portable across shells and have certain behaviours which can make them problematic to work with, there will be an entire chapter dedicated to alisases coming up in the series.

So What?

So we now hopefully have a greater understanding of the variety of shell commands. Not all commands are executables, not all of the commands we think are executables necessarily are, and some commands might be more sophisticated.

As a shell user, the key things to remember are:

Executables are ‘safe’ - they are programs your system can use; your shell just calls out to them.
Builtins are very shell-specific and usually control the shell itself
Functions are powerful ways to write logic but will normally be shell-specific.
Aliases are conveniences for human operators, but only in the context of an interactive shell.

To find out how a command is implemented, just use the type -a command:

$ type -a cat
cat is /bin/cat

More than You Need to Know

OK, for the masochistic few, you might be wondering about all of the other commands and utilities you may have seen that can tell you about programs and commands:

what
whatis
which
whence
where
whereis
command
type

A lot of these are legacy and should be avoided, but for completeness sake, we’ll go through them.

`what`

what reads out special metadata embedded in a program, generally used to identify the version of source code it was built from:

$ what /bin/ls
/bin/ls
 Copyright (c) 1989, 1993, 1994
 PROGRAM:ls PROJECT:file_cmds-272.220.1

There should be almost no circumstance in which you need to use it in your day-to-day work, but you might come across it if you meant to type whatis.

`whatis`

whatis searches a local help database for text. This can be useful in tracking down manual pages:

$ whatis bash
bash(1) - GNU Bourne-Again SHell
bashbug(1) - report a bug in bash

But I can’t imagine it will be a regularly used tool by most users.

`which`

which will search your $PATH to see whether an executable can be found. With the -a flag, it will show all results.

$ which -a vi
/usr/local/bin/vi
/usr/bin/vi

which originated in csh. It remains on many systems for compatibility but in general should be avoided due to potentially odd behaviour⁴.

`whence`

whence was added to the Korn shell. You are unlikely to use it unless you are on systems using ksh. zsh also has this command, but it should be avoided and considered non-standard.

% whence brew
/usr/local/bin/brew

`where`

This is a shell builtin that can provide information on commands, similar to type:

% where ls
ls: aliased to ls -G
/bin/ls

However, type should be preferred, as it is more standard.

`whereis`

whereis is available on some systems and generally operates the same as which, searching paths for an executable:

% whereis ls
/bin/ls

Again, type should be preferred for compatability.

`command`

command is defined in the POSIX standard, so should be expected to be present on most modern systems. Without arguments, it simply executes a command. With the -v argument, you get a fairly machine-readable or processable response; with the -V argument, you get a more human readable response:

% command -v ls
alias ls='ls -G'
% command -V ls
ls is an alias for ls -G

command can be useful in scripts, as we will see in later chapters.

`type`

type is part of the Unix standard and will be present in most modern systems. As we’ve already seen, it will identify the type of command as well as the location for an executable:

% type -a ls
ls is an alias for ls -G
ls is /bin/ls

This command can also be used to only search for paths:

% type -p ls
ls is /bin/ls

Summary

In summary, avoid anything that starts with ‘w’! These are legacy commands, generally needed only when working on older Unix machines. type or command should be used instead.

Footnotes

We will cover permissions and modes in later chapters. ↩︎
Why these names and locations? It’s a long story. The best place to start if you are intersted is the Filesystem Hierarchy Standard. ↩︎
chmod changes the mode of a file; +x means ‘add the executable bit’. This tells the operating system the file can be executed. ↩︎
Stack Exchange: Why not use “which”? What to use then? ↩︎

Effective Shell Part 6: Everything You Don't Need To Know About Job Control

Mon, 10 Jun 2019 08:26:33 +0000

Job control is a feature of most shells, which is generally not particularly intuitive to work with. However, knowing the basics can help prevent you from getting yourself into a tangle, and can from time to time make certain tasks a little easier.

In this chapter, we’ll look at the main features of job control, why it can be a problematic, and some alternatives.

What Is Job Control?

Let’s start with an example. I am building a simple web page. It has one index.html file, one styles.css file, and one code.js file. The index.html file looks like this:

<html>
 <head>
 <title>My New Project</title>
 <link rel="stylesheet" type="text/css" href="styles.css">
 <script src="code.js"></script>
 </head>
 <body>
 <!-- Snip... -->
 </body>
</html>

Opening the file in a browser doesn’t quite work, as it won’t load the code or the styles. We need a web server to serve styles and code.

A super-useful one-liner to run a web server on any machine with Python installed is:

python -m SimpleHTTPServer 3000

In fact, this is so useful that I normally alias this command, so that I can just type serve. We’ll see aliases in a later chapter.

For now, if we run this command (you can get the three sample files here if you want to try this yourself), then we can open the webpage in a browser, with the styles and code loaded:

We can also see that the server has served the HTML, JavaScript, and CSS files:

All well and good so far.

The Problem

Let’s say we want to now continue using our shell, maybe to edit the website with a terminal editor like Vim or Emacs, or we want to zip up the site, or just run any shell command¹.

We have a problem. The python process is still running - it’s serving the website. Our shell is essentially useless, until we stop the server. See what happens when I try to edit a file:

In the example above, I try to run vi, but nothing is happening. Standard input is not being read by the server and not being interpreted by the shell.

I have to kill the server by hitting Ctrl+C (which sends a SIGINT² - we’ll see more about signals later), clear my screen to get rid of all of the error messages, then start again.

This is obviously not optimal. Let’s look at some solutions.

Solution 1: Start the Server in the Background

In most shells, you can run a command and instruct the shell to run it in the background. To do this, you end the line with an ampersand. Here’s how the example would look in this case:

By ending the command with an & ampersand symbol, we instruct the shell to run the command as a background job. This means that our shell is still functional. The shell has also notified us that this command is running as a background job with a specific job number:

% python -m SimpleHTTPServer 3000 &
[1] 19372

In slightly obtuse language, the shell has informed us that it has started a job in the background, with job number 1 and that this job is currently handling the process with ID 19372.

The ampersand solution is a fairly common pattern used in day-to-day work.

Solution 2: Move the Server to the Background

Let’s say you forgot to start the command in the background. Most likely in this case you’d kill the server with Ctrl+C and then start it again with the & option. However, what if this was a large file download or a task you didn’t want to abort?

In the example below, we’ll move the job to the background:

The process is currently in the foreground, so my shell is inactive. Hitting Ctrl+Z sends a ‘suspend’ signal to the process³, pausing it and moving it to the background.

Let’s dissect this:

% python -m SimpleHTTPServer 3000
Serving HTTP on 0.0.0.0 port 3000 ...
127.0.0.1 - - [03/Jun/2019 13:38:45] "GET / HTTP/1.1" 200 -
^Z
[1] + 21268 suspended python -m SimpleHTTPServer 3000

The shell echos as I type, so we see ^Z (i.e., the Ctrl+Z chord I entered). The shell responds by moving the process into a background job and suspending it.

The key here is that it is suspended. The process is paused. So the web server is no longer serving. If you are following with the sample, reload your browser. The webpage fails to load, as the server process is not able to respond to requests.

To continue the job, in the background, we use the bg (‘background’) command, with a job identifier (which always starts with a % symbol - we’ll see why soon) to tell the shell to continue the job:

% bg %1
[1] + 21268 continued python -m SimpleHTTPServer 3000

The shell lets us know the job is being continued, and if we load the webpage again, the content is shown as expected.

As a final check, we run the jobs command to see what jobs the shell is running:

% jobs
[1] + running python -m SimpleHTTPServer 3000

And there you have it - our server is running as a background job. This is exactly what we would see if we run jobs after starting the server with an & at the end. In fact, using an & is perhaps an easier way to remember how to continue a suspended job:

% %1 &
[1] + 21268 continued python -m SimpleHTTPServer 3000

In the same way ending a command with & runs it in the background, ending a job identifier with & continues it in the background.

There is at least one more way to move a job to the background⁴, but I have not yet found it useful in any scenarios, and it is overly complex to explain. See the footnote for details if you are interested.

Moving Background Jobs to the Foreground

If you have a job in the background, you can bring it back to the foreground with the fg (‘foreground’) command. Let’s show the jobs, with the jobs command:

% jobs
[1] + running python -m SimpleHTTPServer 3000

Here I have a background job running a server. Any one of the following commands will bring it back to the foreground:

fg %1 # Explicitly bring Job 1 into the foreground

%1 # ...or in shorthand, just enter the job id...

fg # ...if not given an id, fg and bg assume the most recent job.

Now the job is in the foreground, and you can interact with the process again however you like.

Cleaning Up Jobs

You might realise you cannot continue what you are doing because an old job is still running. Here’s an example:

I tried to run my web server, but there was still one running as a background job. The server failed to start because the port is in use.

To clean it up, I run the jobs command to list the jobs:

% jobs
[1] + suspended python -m SimpleHTTPServer 3000

There’s my old web server. Note that even though it is suspended, it’ll still be blocking the port it is serving on⁵. The process is paused, but it is still holding onto all of the resources it is using.

Now that I know the job identifier (%1 in this case), I can kill the job:

% kill %1
[1] + 22843 terminated python -m SimpleHTTPServer 3000

This is why job identifiers start with a percentage sign! The kill command I have used is not a special job control command (like bg or fg). It is the normal kill command, which terminates a process. But shells that support job control can normally use a job identifier in place of a process identifier. So rather than working out what the process identifier is that I need to kill, I can just use the job identifier⁶.

Why You Shouldn’t Use Jobs

Avoid jobs. They are not intuitive to interface with, and they suffer from some serious problems.

The most obvious one is that all jobs write to the same output, meaning you can quickly get garbled output like this:

This is what happens when I run a job, which just outputs text every second. It’s in the background, but it’s printing all over my commands. Even running the jobs command to try and find the job to stop it is difficult.

Input is even more complex. If a job is running in the background, but requires input, it will be silently suspended. This can cause confusion.

Jobs can be used in scripts but must be done so with caution and could easily confuse a consumer of the script if they leave background jobs hanging around, which cannot be easily cleaned up⁷.

Handling errors and exit codes for jobs can be problematic, causing confusion, poor error handling, or overly complex code.

How to Escape Jobs

If there are two things to take away, it would be this:

If you have started running a command in the foreground, and you don’t want to stop it and would rather move it to the background, hit Ctrl+Z. Then Google “job control”.

And:

If you think there is a job running in the background, and it is messing with your screen, type fg to bring it to the front and kill it with Ctrl+C. Repeat as needed!

In either case, if you need to do something more subtle, you can return to this reference. But the first command should allow you to get your shell back while you work out how to continue the job, and the second should kill a background job that is messing with your screen.

Alternatives to Jobs

If you are using any kind of modern terminal such as iTerm, Terminal or the GNOME Terminal, just open a new tab or split! Much easier.

The benefit to this is that each tab gets its own standard input and output, so there’s no risk of overwriting. And of course you can hide/reveal/rearrange the tabs however you like.

The traditional alternative to a job for an operator who simply wants more than one thing going on at once would be a terminal multiplexer, such as screen or tmux:

Multiplexers work in a very similar way to a modern graphical terminal - they manage many shell instances. The benefits to a modern terminal, such as iTerm, is that you have a very intuitive GUI and lots of features.

The benefits to a multiplexer are that you can run them over SSH sessions to manage complex operations on remote machines and that they run a client-server model, meaning many people can work with many multiplexed processes (and they can persist beyond sessions).

My personal preference is both - I use a modern terminal and run everything inside it in tmux. We’ll look at both of these options in later chapters.

Quick Reference

You might find that jobs are useful, or you might find that they are not. Either way, here’s a quick reference of some common commands:

Command	Usage
`command &`	Run the command as a background job.
`<Ctrl+Z>`	Move the current process into a background job, suspended.
`jobs`	List all jobs.
`fg %1`	Move background job number 1 into the foreground.
`bg %1`	Continue background job number 1.
`kill %1`	Terminate job number 1.
`wait %1`	Block until job number 1 exits.

If you want to find out more about the gory details of jobs, the best place to start is the Bash Manual - Job Control Section, or the ‘Job Control’ section of your preferred shell’s manual.

I hope you found this useful, and, as always, please leave comments, questions or suggestions below!

Footnotes

If you are not a heavy shell user, this might seem unlikely. But if you do a lot of work in shells, such as sysadmin, devops, or do your coding from a terminal, this happens all the time! ↩︎
Signals like SIGINT, SIGKILL, SIGTERM and so on will be covered in a later chapter. ↩︎
Technically, SIGTSTP - which is ‘TTY stop’. If you have always wondered about the ‘TTY’ acroynm, check the previous chatper, Interlude: Understanding the Shell. ↩︎
The alternative method is to use Ctrl+Y, which will send a delayed interrupt, which will continue to run the process until it tries to read from stdin. At this point, the job is suspended and the control given to the shell. The operator can then use bg or kill or fg to either move to the background, stop the process, or keep in the foreground as preferred. See: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Job-Control ↩︎
Another super-useful snippet: lsof -i -P -n | grep 8000 to find any process that has a given port open. Another one for the aliases chapter! ↩︎
There are times this is needed. If a job runs many processes - for example, by running a pipeline - the process identifier will change as the command moves from one stage of the pipeline to the next. The job identifier will remain constant. Remember, a job is a shell command, so could run many processes. ↩︎
To see how bad this can be, create a script that starts jobs, then run it. Then run the jobs command to see what is running. The output might surprise you! ↩︎

Effective Shell Interlude: Understanding the Shell

Tue, 21 May 2019 09:22:05 +0000

This is the first ‘interlude’ in my Effective Shell series. These interludes give some background, history or more flavour to some of the topics.

This one should be high-level enough for even non-technical readers to enjoy (or at least understand!). I’ve tried to make sure any term that might be unfamiliar is described in a footnote¹. For the more technical reader, it provides an important grounding on some of the key concepts relating to shells and how they work.

Introduction for the Non-Technical Reader

It might come as a surprise that many technical computer users (programmers, data scientists, systems administrators etc) spend a lot of time using an interface which looks like it’s from the sixties:

If you work with technologists, you might have seen them using an interface like this. This kind of simple, text-based interface is called a shell, and it has been a common way to interface with computers ever since the first screens and keyboards were created.

Given how much computing has advanced, why would people use such an interface? Just look at how much the Windows operating-system has changed over the last three decades:

(By Source (WP:NFCC#4), Fair use, https://en.wikipedia.org/w/index.php?curid=58853841)

Why would people choose to use such an archaic interface as a shell?

Typing is fast: A skilled shell user can manipulate a system at dazzling speeds just using a keyboard. Typing commands is generally much faster than exploring through user interfaces with a mouse
Shells are programmable: Users will often being programming as they work in a shell, creating scripts to automate time-consuming or repetetive processes
Shells are portable: A shell can be used to interface to almost any type of computer, from a mainframe to a Raspberry Pi, in a very similar way.

Not all technical users will use a shell regularly, but there are many who will spend the bulk of their time in such an interface. It is such a crucial skill to be able to operate one effectively that I have been writing this series primarily to show ways to be more efficient with this kind of interface.

Introduction for the Technical Reader

You may be familar with the shell, but it can be useful to understand some of the surrounding concepts in detail. How does a shell differ from a terminal? What is a tty? How do shells really work? Hopefully as you read this article you’ll discovery something that you didn’t know about shells.

Let’s Get Started!

To understand what shells, terminals, command-prompts and so on are and how they relate, we need to start with the basics: how a modern computer works!

A Computer in a Nutshell

The diagram below shows a simplified view of a typical computer:

Already there’s a lot going on.

Your computer is going to have a CPU² and memory³, and almost certainly a network adapter⁴ and display adapter⁵. Most computers will have at least one hard disk. For home PCs, there’ll also likely be a bunch of peripherals, such as a mouse, keyboard, printers, flash drives, webcams and so on.

The Operating System

The operating system is the piece of software installed on a computer that can interface with the hardware. Without hardware, such as a CPU, memory, a network adapter, a graphics card, disk drives and so on, there’s not much that you can do with the computer. The operating system is the primary interface to this hardware. No normal programs will talk to hardware directly - the operating system abstracts this hardware away and provides a software interface to it.

The abstraction the operating system provides is essential. Developers don’t need to know the specifics of how to work with individual devices from different vendors; the operating system provides a standardised interface to all of this. It also handles various tasks such as making sure the system starts up properly.

The operating system is generally broken down into two parts - the kernel and user space:

Let’s look at these in more detail.

The Kernel

This is the part of the operating system that is responsible for the most sensitive tasks: interfacing with physical devices, managing the resources that are available for users and programs, starting up the various systems that are needed, and so on.

Software running in the kernel has direct access to resources, so is extremely sensitive. The kernel will balance resources between the programs in user space, which we’ll look at shortly. If you’ve ever had to install ‘drivers’, these are examples of pieces of software that will run in the kernel - they’ll have direct access to a physical device you’ve installed, and expose it to the rest of the software on the computer.

Why ‘kernel’? The kernel is the soft, edible part of a nut or seed, which is surrounded by a shell. Below you can see a walnut - the kernel is the soft bit in the middle, and the shell surrounds and protects it. This is a useful metaphor that is used for parts of a computer.

(By Kkchaudhary11 - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=49069244)

The operating system kernel really is the core of the operating system. It’s such a sensitive area of the operating system that we actually want to avoid running software in it if possible⁶. And that is where user space comes in.

User Space

The vast majority of programs run in ‘user space’ (also commonly called ‘user land’).

When a program starts, the kernel will allocate it a private segment of memory and provide limited access to resources. The program is given access to a library of functions by the operating system, which it can use to access resources such as files, devices and so on. Programs in user space are essentially in sandboxes, where there is a limit to how much damage they can do.

For example, a program running in user space can use the standard fopen function, which is provided on almost every operating system as part of the C Standard Library. This allows a program to attempt to open a file. The operating system will make a decision on whether the program is allowed to open the file (based on things such as permissions, where the file is and so on) and then, if it is OK with the call, will give the program access to the file. Under the hood, this ‘user space’ call translates to a system call in the kernel.

Now that the key components have been introduced, we can look at the shell. The name should come as no surprise, as it is a wrapper or outer layer to the operating system (which itself contains the sensitive nugget of the kernel).

The Shell

So what is the shell? The shell is just a general name for any user space program that allows access to resources in the system, via some kind of interface.

Shells come in many different flavours but are generally provided to aid a human operator in accessing the system. This could be interactively, by typing at a terminal, or via scripts, which are files that contain a sequence of commands.

For example, to see all of the files in a folder, the human operator could write a program in a language such as C, making system calls to do what they want. But for day-to-day tasks, this would be repetitive. A shell will normally offer us a quick way to do that exact task, without having to manually write a program to do it.

Here’s an example, where a shell is being used to show the ‘png’ images in the folder I am working in⁷:

So a shell is a user-space program to interface with the computer. But there a few more moving parts than just a shell we are seeing in the image above. There are different types of shells, there are terminal programs, and there are the programs or commands that the shell calls (in the example above, tree is a program). Let’s pick these apart.

Here’s a diagram that more accurately shows what is going on:

We’ve introduced a few new things here. There’s a user, who is interfacing with a terminal, which is running a shell, which is showing a command prompt. The user has written a command that is calling a program (in this case, the tree program).

Let’s dissect this bit by bit.

The Terminal

We’re not directly interacting with the ‘shell’ in this diagram. We’re actually using a terminal. When a user wants to work with a shell interactively, using a keyboard to provide input and a display to see the output on the screen, the user uses a terminal.

A terminal is just a program that reads input from the keyboard, passes that input to another program (normally a shell), and displays the results on the screen. A shell program on its own does not do this - it requires a terminal as an interface.

Why the word terminal? This makes sense when you look at how people interfaced with computers historically. Input to a computer might be through punch cards, and output would often be via a printer. The Teletype Termimal⁸ became a common way for users to interface with computers.

(Photograph by Rama, Wikimedia Commons, Cc-by-sa-2.0-fr, CC BY-SA 2.0 fr, https://commons.wikimedia.org/w/index.php?curid=17821795)

At this time, computers were very large, complex, and expensive machines. It was common to have many terminals connected to a single large machine (or ‘mainframe’), or a few terminals that people would share. But the terminal itself was just a human interface to the operating system. A more modern terminal would be something like an IBM 3486:

(By ClickRick - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=6693700)

This is a very small computer in its own right but still basically just a dumb screen and keyboard connected by a cable to a larger mainframe computer in another location.

This mechanism is still very much the case today. When I want to work with a computer in a data centre, I don’t go and find the machine, plug in a keyboard and a display and directly interface to it. I run a terminal program on my computer to provide access to the remote machine. My terminal program allows me to use my keyboard and display to work with a remote machine - all via a secure shell - which is a secured-shell connection over a network.

So terminals in many ways are quite simple - they are interfaces. But because they are quite simple programs, we can’t do much with them. So normally, the first thing that a terminal program will do is run a shell program - a program that we can use to operate the computer.

There’s nothing special about terminals - anyone can write a program to operate as a terminal, which is why you will see many different terminals around. Examples are the standard ’terminal’ app for MacOS X, the gnome-terminal for Linux, and iTerm2 and Hyper. There’s a bunch of screenshots of different setups at the end of the article.

Back to the Shell

Now that we’ve described the terminal, we can go back and look at the shell in detail.

The shell is the program that is going to take input from somewhere and run a series of commands. When the shell is running in a terminal, it is normally taking input interactively from the user. As the user types in commands, the terminal feeds the input to the shell and presents the output of the shell on the screen.

A shell program can also take input from files; these files will then generally be ‘shell scripts’. This might be used to run automated operations, such as cleaning up certain folders when a computer starts.

Shells can write output to files or other locations, and so on. You can run a shell program outside of a terminal - you just won’t be able to interface with it using a keyboard or display. And in fact, lots of operations happen in this way: automated scripts, startup tasks, installers and so on.

So what else does a shell do? Most of the features are related to helping human operators work with the system more efficiently.

Quickly enter commands, see the history of commands and quickly restructure commands (see Effective Shell - Navigating the Command Line)
Navigate through the file system, moving from folder to folder (see Effective Shell - Move Around!), which makes it easier for an operator to navigate the file system.
Chain the output of commands together - for example, taking the output of one basic program, such as the tree program we saw, and writing it to a file (see Effective Shell - Understanding Pipelines)
Offer a programming language, allowing the operator to perform more complicated tasks (see Effective Shell - Basic Shell Scripting)

And a lot more! In fact, that’s what the whole Effective Shell series is about - how to get the most from these powerful programs, particularly for those who use them regularly.

The Command Prompt or Command Line

The last part of the diagram, which we haven’t covered yet, is the command prompt.

When a shell is running in terminal, it knows that a human operator will be interfacing with it. So to make sure that the operator has some kind of visual hint that they have to enter commands, the shell will output some kind of prompt.

I’ve included a set of screenshots at the end of the article, just after this section, and you can see how some different command prompts look.

Note that shells don’t have to use command prompts - if you use a shell program to execute a script, there will be no command prompt. Shells only show a prompt when they know they are being used interactively. Many programs which allow a user to operate interactively will show a command prompt.

Shell command prompts can be customised, so they will often look different from machine to machine (for more details, see Effective Shell - Customising the Command Line). Below is an example that shows a lot of technical information. This is from the highly popular oh-my-zsh framework for the ‘Z Shell’ shell, which is very popular among developers:

*(Source: https://ohmyz.sh/)

Shell Commands and Different Shells

A lot of the ‘commands’ in a shell, such as cat (which shows the contents of a file), are actually just simple programs, which will interface with the kernel. No matter what shell you use, these commands will behave the same way, because really all you are doing is calling another progam.

Some commands, such as cd (change directory), are built into the shell. Some commands are functions that have been defined, or aliases to other commands (for more details on commands, see Effective Shell - Commands). Commands will often differ between shells.

Not all shells are created equal - anyone can write a shell program, maybe creating a simple interface to the computer or a highly complex one with many features. In fact, a later article in this series will look at the geneology of the most common shells.

On most Unix-like systems, the default shell is a program called bash, which stands for " Bourne Again Shell" (the name and history around it will be discussed at length in the later article). But there are many other shells: the C Shell, the Korn Shell, Z Shell and Fish, just to name just a few.

Users and administators can configure what shell they like to use. When a terminal opens, it will immediately start the user’s preferred shell program. It is possible to change this. Different users will have different preferences, given that shells offer varying features. This can cause complexity when working with systems, as we cannot always expect every user to have the same shell, or even for the same shell to be set up consistently, as they can be extensively customised.

Let’s review the earlier diagram again:

We can see the real internals of what is going on in this “Terminal -> Shell -> Program” chain in the diagram above quite easily.

Try the command pstree -psa $$ in a shell⁹:

The first systemd process is the primary process for the OS - it is process number 1, which initialises everything else. The second systemd process is the process that is running the interface for my user. We can ignore these for now; they are internals to how the operating system boots and starts processes.

What is interesting is that we can see a terminal (the gnome terminal), which has started my preferred shell (which is zsh), which is running a command (the program pstree). Here we can see the exact chain as shown in the diagram earlier.

That’s a Wrap!

These are the key technologies and concepts that surround a shell.

If you are interested in more technical details of working with shells, then my Effective Shell series goes into these topics in depth. The goal of this series is to help teach techniques that making working with shells more efficient.

To close the article, below are some examples of different terminals, shells, command prompts and so on.

Example: iTerm 2 / tmux / zsh

In this example, we have:

A MacOS operating system
iTerm2 as the terminal program
tmux running as a ’terminal multiplexer’ (see Effective Shell: Terminal Multiplexers)
zsh (Z Shell) as the shell program, using ‘oh my zsh’, which is easily recognised by the % sign in the command prompt.
A customised command line, which shows the user and folder on one line, with only the % symbol below, to leave lots of space for the input commands¹⁰.

Example: Bash

In this example, we have:

A Linux operating system (Ubuntu 14)
The gnome terminal
bash as the shell
In the second screenshot, the user has ‘root privileges’, and to indicate this, bash helpfully changes the default command prompt from a dollar sign to a hash sign

Example: Windows Explorer

In this example, we have:

The Windows 10 operating system
No terminal
The explorer.exe program showing us a graphical shell

This looks different from previous examples. The program, which shows the familiar Windows interface, explorer.exe, is in fact a shell as well, offering interactive access to the operating system and computer resources. The bulk of the Windows APIs to interact with this interface are in the Shell Library. I also maintain a popular library for building extensions to the graphical Windows shell - sharpshell.

Example: Windows Command Prompt

In this example, we have:

The Windows 10 operating system
The command prompt terminal and shell

In Windows, the terminal and shell are combined into a single cmd.exe program. There’s an excellent article on the internals - Microsoft DevBlogs: Windows Command-Line: Inside the Windows Console

Example: Windows PowerShell

In this example, we have:

The Windows 10 operating system
The PowerShell terminal

PowerShell is an improvement on the ‘command prompt’ program that was originally used in Windows, offering much more functionality for scripting and other modern shell features.

Example: Windows Subsystem for Linux (WSL)

In this example, we have:

The Windows 10 operating system
The Bash.exe program

This screenshot, from MSDN: Frequently Asked Questions about Windows Subsystem for Linux shows Bash running in Windows. This is a relatively new feature at the time of writing, allowing Windows users to use a Linux interface to the PC. This is a feature that may become increasingly valuable, as in general it is challenging to write shell code that can run on Windows and Unix-like systems.

If you enjoyed this article, please do share it! Feel free to include suggestions, improvements or corrections in the comments below.

Useful References

A simple Linux kernel module, showing how basic kernel programming works in Linux: github.com/dwmkerr/linux-kernel-module
How Linux Works - Brian Ward
StackExchange: What is the exact difference between a ’terminal’, a ‘shell’, a ’tty’, and a console?
Microsoft: Inside the Windows Console

Footnotes

I’d be fascinated to know if this is at all interesting to less technically inclined people, so please do go ahead and let me know in the comments! ↩︎
CPU: central processing unit. This is the chip in the computer that does most of the work (which after many layers of abstraction eventually becomes arithmetic and sending simple instructions to other places). ↩︎
Memory is the ‘working space’ where the state of your system is stored. If you are writing a document, the text lives in memory, until you save it, when it then gets written to a hard drive. Memory is ephemeral - everything is gone when you turn off the power to it. ↩︎
This is the part of your computer that knows how to do things like connect to a WiFi network, or has a network socket you might plug a network cable into. ↩︎
This is the part of your computer you plug the screen into. ↩︎
This is because a mistake in Kernel Mode programs can have disasterous effects. It could access any files, no matter who they belong do, control the hardware, install more software - almost anything. Errors in this code can cause terrible issues (like the infamous Windows ‘blue screen of death’), and malicious code in the kernel essentially has full access to not only all your data but also your webcam, network adapter and so on. ↩︎
As an aside, if you are curious about the visual style of my setup or customisations that have been made, everything in my setup is available online on my ‘dotfiles’ repo - github.com/dwmkerr/dotfiles. ↩︎
And that’s where the ‘TTY’ acronym you will see sometimes comes from. Enter the ps command, and you’ll actually see the TTY interface each process is attached to. This is a topic that will come up later in the series. ↩︎
$$ is a Bash internal variable. These will also be covered in a later article in the series. ↩︎
Feel free to see my dotfiles to configure a similar setup for yourself. ↩︎

Effective Shell 4: Move Around!

Mon, 11 Mar 2019 09:02:00 +0000

This is the fourth part of my Effective Shell series, a set of practical examples of ways to be more efficient with everyday tasks in the shell or at the command line.

In this article we’ll look at the key elements of navigation in the shell.

Getting Comfortable Moving Around

You might already spend a lot of time in the shell, running various command line programs or using tooling for development projects or operational tasks. But you might also still switch back to a more visual paradigm for working with files, directories and resources.

Being able to perform everyday file and folder manipulation tasks directly from the shell can really speed up your workflow. Let’s look at some common tasks and see how we can work with them in the shell. Along the way we’ll also introduce some of the most frequently used tools and commands to work with the filesystem.

Where Am I?

The first command to become familiar with is pwd (‘print working directory’). This command will echo the current absolute path. You can also use the $PWD environment variable:

$ pwd
/Users/dave/repos/github/dwmkerr/effective-shell

$ echo $PWD
/Users/dave/repos/github/dwmkerr/effective-shell

Depending on your shell, or your command-line setup (which we will discuss in a later chapter), you might also see your working directly on the command-line.

Changing Directory

Most likely one of the most familiar commands out there, the cd or chdir function changes the current directory:

$ pwd
/Users/dave/repos/github/dwmkerr/effective-shell

$ cd

$ pwd
/users/dave

$ cd -
~/repos/github/dwmkerr/effective-shell

$ pwd
/Users/dave/repos/github/dwmkerr/effective-shell

$ cd ~

$ pwd
/users/dave

Here we can see that running cd with no parameters moves to the users ‘home’ directory. This directory is always available in the $HOME environment variable.

Running cd - will switch back to the previous directory — this is very useful if you want to quickly jump somewhere and then back again.

You can use ~ as an alias for the home directory, allowing you to quickly move to personal folders, with commands such as cd ~/Downloads.

Most commonly, you will specify a path when changing directory. This can be a fully qualified path, or it can be a relative path:

$ cd /dev

$ cd ~/repos

$ cd ./github

You can use the special link .., which is a folder that points to the parent directory to move ‘upwards’:

$ pwd
/Users/dave/repos/github/dwmkerr/effective-shell

$ cd ../../

$ pwd
/Users/dave/repos/github

Exploring a Directory

Once we are in a directory, we will often want to see the contents. The ls (“list directory contents”) command is useful here:

$ pwd
/Users/dave/repos/github/dwmkerr/effective-shell

$ ls
1-navigating-the-command-line LICENSE
2-clipboard-gymnastics README.md
3-getting-help sed.1
4-moving-around

By default, the ls command will list the files and directories. We can show more information with the -l (“long format”) flag:

$ ls -l
total 48
drwxr-xr-x 6 dave staff 192 Mar 5 16:01 1-navigating-the-command-line
drwxr-xr-x 5 dave staff 160 Oct 10 2017 2-clipboard-gymnastics
drwxr-xr-x 4 dave staff 128 Dec 19 2017 3-getting-help
drwxr-xr-x 3 dave staff 96 Mar 7 15:39 4-moving-around
-rw-r--r-- 1 dave staff 1066 Jun 10 2017 LICENSE
-rw-r--r-- 1 dave staff 561 Mar 7 15:30 README.md
-rw-r--r-- 1 dave staff 15707 Mar 5 16:01 sed.1

Now we can see the permissions, the link count (which is rarely particularly useful and varies from platform to platform), the owner, the group, the size and the modification date (as well as the name).

We can make the sizes more human readable, and sort by size with a few more flags -h (“human readable”) and -s (“sort by size”):

$ ls -lhS
total 48
-rw-r--r-- 1 dave staff 15K Mar 5 16:01 sed.1
-rw-r--r-- 1 dave staff 1.0K Jun 10 2017 LICENSE
-rw-r--r-- 1 dave staff 561B Mar 7 15:30 README.md
drwxr-xr-x 6 dave staff 192B Mar 5 16:01 1-navigating-the-command-line
drwxr-xr-x 5 dave staff 160B Oct 10 2017 2-clipboard-gymnastics
drwxr-xr-x 4 dave staff 128B Dec 19 2017 3-getting-help
drwxr-xr-x 3 dave staff 96B Mar 7 15:39 4-moving-around

There are lot of options for ls. Check the chapter Getting Help for some tips on how to get more information on a command!

Managing the Directory Stack

You might find that you want to move to a number of directories, then return to where you started. This can be particularly useful when scripting. You can use the pushd (“push onto directory stack”) and popd (“pop from directory stack”) commands to add or remove directories from the stack:

$ pwd
/Users/dave/repos/github/dwmkerr/effective-shell

# OK - I'm writing my article at the moment, but want to check my downloads, and come back shortly...
# Move to the downloads folder...

$ ls
aws-nuke-v2.8.0-darwin-amd64

# OK cool - the tool I was downloading has arrived, let's use it...
cd aws-nuke-v2.8.0-darwin-amd64
./aws-nuke

# Now I want to go back to my article...
$ popd
~/Downloads ~/repos/github/dwmkerr/effective-shell

~/Downloads
$ popd
~/repos/github/dwmkerr/effective-shell

In this case, using cd - would not be sufficient — that would just switch us from the aws-nuke folder to Downloads and back again. But by using the directory stack we can save where we are, move, and then ‘pop’ our way back to where we started.

Auto-Completion

Pressing tab when using commands like cd will generally show an auto-completion menu:

$ cd ~/repos/ # press 'tab' now...
github/ gitlab/ local/ scratch/

Pressing tab again will cycle through options, and shift-tab will cycle backwards. Enter will select an option, escape (or Ctrl-C) will cancel.

Some shells, such as zsh, support even more advanced auto-completion. For example, we can auto-complete to fill in partially specified directory names:

% cd ~/r/g/d/e # press tab now...

% cd ~/repos/github/dwmkerr/effective-
effective-container-engineering/ effective-shell/

Auto-completion is generally very shell specific. We’ll look more into the different shells that are available in later chapters. But in general, if you are uncertain, pressing tab will often show a sensible set of options.

That’s It!

This is a small chapter, but an important one. Later on, as we start to do more file and system manipulation from the shell, moving and copying files and so on, we will build on these concepts. But it is critical to first know the basics of how to move around the filesystem with the shell.

A portable and magic-free way to open Pull Requests from the Command Line

Wed, 10 Oct 2018 09:17:26 +0000

This little bash snippet will let you open a GitHub or GitLab pull request from the command line on most Unix-like systems (OSX, Ubuntu, etc), without using any magic libraries, ZSH tricks or other dependencies.

tl;dr download the gpr.sh gist.

Here’s how it looks in action OSX:

And Ubuntu:

The script is available as the gpr.sh gist. You can also find it in my dotfiles, in the git.sh file.

The Script

Here’s the script in its entirety:

# Colour constants for nicer output.
GREEN='\033[0;32m'
RESET='\033[0m'

# Push the current branch to origin, set upstream, open the PR page if possible.
gpr() {
 # Get the current branch name, or use 'HEAD' if we cannot get it.
 branch=$(git symbolic-ref -q HEAD)
 branch=${branch##refs/heads/}
 branch=${branch:-HEAD}

 # Pushing take a little while, so let the user know we're working.
 echo "Opening pull request for ${GREEN}${branch}${RESET}..."

 # Push to origin, grabbing the output but then echoing it back.
 push_output=`git push origin -u ${branch} 2>&1`
 echo ""
 echo ${push_output}

 # If there's anything which starts with http, it's a good guess it'll be a
 # link to GitHub/GitLab/Whatever. So open it.
 link=$(echo ${push_output} | grep -o 'http.*' | sed -e 's/[[:space:]]*$//')
 if [ ${link} ]; then
 echo ""
 echo "Opening: ${GREEN}${link}${RESET}..."
 python -mwebbrowser ${link}
 fi
}

How It Works

Blow-by-blow, let’s take a look.

# Colour constants for nicer output.
GREEN='\033[0;32m'
RESET='\033[0m'

To make colouring console output easier, we create strings with the escape code required to set the ‘green’ colour, and reset the text colour.

gpr() {
 # Get the current branch name, or use 'HEAD' if we cannot get it.
 branch=$(git symbolic-ref -q HEAD)
 branch=${branch##refs/heads/}
 branch=${branch:-HEAD}

Now we define the gpr (Git Pull Request) function. We’ll need to push the current branch, so we need to get the current branch name. There’s plenty of discussion on how this works on Stack Overflow: How to get the current branch name in Git. Essentially we just get the symbolic name for the head of our current branch, which will be something like this:

refs/heads/my-new-branch

We then use Bash substring removal to rip out the ref/heads/ part. If we have no branch (for example, we are detached) we just use HEAD a the branch name.

Next we have this:

 # Pushing take a little while, so let the user know we're working.
 echo "Opening pull request for ${GREEN}${branch}${RESET}..."

 # Push to origin, grabbing the output but then echoing it back.
 push_output=`git push origin -u ${branch} 2>&1`
 echo ""
 echo ${push_output}

We’ve previously defined some strings which include the escape codes to colour terminal output. Now we just show the user the branch we’re going to push, push it and then store all of the output in the push_output variable.

The 2>&1 idiom is a common one. This simply makes sure we put all stderr output (which is always file descriptor 2) into stdout (which is always file descriptor 1). This means whether the program writes output to stdout or stderr, we capture it. There’s a nice write-up on this in the blog post ‘Understanding Shell Script’s idiom: 2>&1 ’.

The output from Git push will be dependent on the Git server being used. For GitHub it’ll look like this:

remote:
remote: Create a pull request for 'feat/doc-cleanup' on GitHub by visiting:
remote: https://github.com/dwmkerr/dotfiles/pull/new/feat/doc-cleanup
remote:
To github.com:dwmkerr/dotfiles
* [new branch] feat/doc-cleanup -> feat/doc-cleanup
Branch feat/doc-cleanup set up to track remote branch feat/doc-cleanup from origin.

Now all we want to do is see if there is any text which starts with http and if there is, then open it. Here’s how we do that:

 # If there's anything which starts with http, it's a good guess it'll be a
 # link to GitHub/GitLab/Whatever. So open it.
 link=$(echo ${push_output} | grep -o 'http.*' | sed -e 's/[[:space:]]*$//')
 if [ ${link} ]; then
 echo ""
 echo "Opening: ${GREEN}${link}${RESET}..."
 python -mwebbrowser ${link}
 fi

This uses grep to rip out everything from http onwards, and the sed to remove any trailing whitespace. If we have found a link, we use python to open it (which is a fairly safe cross-platform solution).

That’s it! When you have a branch ready which you want to push and create a pull request from, just run:

gpr

And the branch will be pushed to origin, and if there is a Pull Request webpage, it’ll be opened.

Prior Art

My colleague Tobias recently shared a nice trick we worked out to open a GitLab merge request - which also now works for GitHub:

git push and directly open PR in Chrome - works for @github & @gitlab 🚀

Here is how to set it up 👉 https://t.co/YfNTmdwTFt #github #gitlab pic.twitter.com/ISE9kVZmw1
— Tobias Büschel (@TobiasBueschel) September 19, 2018

I wanted to be able to use the same trick in Ubuntu and other Linux distros, but realised it relied on oh-my-zsh and assumed OSX with Chrome as the browser, so tweaked it to the above. Thanks Tobi!

Effective Shell Part 3: Getting Help

Tue, 19 Dec 2017 09:05:18 +0000

This is the third part of my Effective Shell series - practical examples of ways to be more efficient with everyday tasks in a shell.

In this article I’ll show you how to quickly get help when working with tools in the shell, without disrupting your flow!

Getting Help is Important!

If you are trying to be more effective when using the shell, it is crucial to know how to quickly look things up.

There’ll be many circumstances where you’ll need to open a browser to search for help, but there’s also a wealth of information only a few keystrokes away. Looking up parameters, checking how to run commads, C library docs or useful information like ASCII charts are available directly in the system.

Before we look at the standard way of accessing documentation on unix-like systems, which is the man command, I’m going to introduce tldr.

Nine times out of ten I get the help I need in a few seconds with tldr, so if you take only one thing away from the article, take the first section. Then if you want to learn more about the system manuals, read on!

tl;dr

Let’s say I need to find and replace some text in a file. I know I can do this with the sed command, but have forgotten the syntax.

All I need to do is run tldr sed:

The first example is exactly what I’m looking for. Now for any more detail than a few basic examples, I’m going to have to go to the manual, but it’s overkill for the basics. Here’s what man sed shows me:

And this is just page one of six! There’s a lot of detail, which is great sometimes, but for a quick lookup, tldr is perfect.

You can install the tldr tool with npm install -g tldr. It’s open source and community maintained.

Now a lot of the time, you are still going to need more help or more detail. For the rest of the article, we’ll dive a bit deeper into man, the system manual pages.

Understanding ‘man’

Most tools you encounter in the shell have manual pages available. Many people will be familiar with the man command to get help on a tool, but let’s take a look in a bit more detail, there’s actually a lot more available than just the documentation for common commands.

Getting help on a command

The most basic way to get help on a command is with man. Here’s an example:

$ man cp
CP(1) BSD General Commands Manual CP(1)
NAME
cp -- copy files
SYNOPSIS
cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ...
target_directory
DESCRIPTION
In the first synopsis form, the cp utility copies the contents of the
source_file to the target_file. In the second synopsis form, the con-
tents of each named source_file is copied to the destination
target_directory. The names of the files themselves are not changed. If
cp detects an attempt to copy a file to itself, the copy will fail.
...

The man command opens the manual for the given tool. These manuals should contain all command line options and details of how to use the tool.

You can scroll up and down through the content with the arrow keys, this is because the information is presented in the shell pager, which is a tool for looking through content which might not easily fit on a screen.

The first thing you might notice is that you can move through the manual pages with the arrow keys.

Manpages are just text files, and man opens them in a pager tool, which is what is providing the keyboard interface to look through the file.

On most systems, the pager will be the less program. There are lots of commands you can use to navigate through files with less, but the bare essentials are:

d - Scroll down half a page
u - Scroll up half a page
j / k - Scroll down or up a line. You can also use the arrow keys for this
q - Quit
/<search> - Search for text
n - When searching, find the next occurrence
N - When searching, find the previous occurrence

There are many other commands, but the set above is normally what I find myself using the most.

If you are interested, you can actually see what your pager is with the command below:

$ echo $PAGER
less

The $PAGER environment variable is used to tell the shell what program to use for paging. More details are found with man man.

You can put any text content into your pager - try this:

ls -al /usr/bin | less

This lists the contents of the /usr/bin folder, piping the output to less so we can easily scroll through it.

There are alternative pagers available (on many Unix-y systems you’ll have less, more and most) but in general you’ll normally get what you need with less.

What’s with the numbers?

You’ll often see tools referred to in manpages with numbers after them. Take a look at man less:

The number is the manual Section Number. The different sections of the manual are documented be found on most unix-like systems in man’s documentation, which you can check by running man man¹. Here’s what you’d get on Ubuntu 16:

| 1 | Executable programs or shell commands | | 2 | System calls (functions provided by the kernel) | | 3 | Library calls (functions within program libraries) | | 4 | Special files (usually found in /dev) | | 5 | File formats and conventions eg /etc/passwd | | 6 | Games | | 7 | Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7) | | 8 | System administration commands (usually only for root) | | 9 | Kernel routines [Non standard] |

We’ll go through the setions in detail shorltly.

You can specifically choose which section of the manual you are looking in by using:

man <section> <search>

You can also get more information about the sections themselves by opening up the intro page. For example:

$ man 1 intro
INTRO(1) BSD General Commands Manual INTRO(1)
NAME
intro -- introduction to general commands (tools and utilities)
DESCRIPTION
Section one of the manual contains most of the commands which comprise...

Why would you do this, and why would you care? A few examples from each section show how this can be quite useful to know about.

Section 1: Programs and Shell Commands

These are programs, probably what you are going to be looking up most regularly! For example, man 1 time shows:

TIME(1) BSD General Commands Manual TIME(1)
NAME
time -- time command execution
SYNOPSIS
time [-lp] utility
DESCRIPTION
The time utility executes and times utility. After the utility finishes, time writes the total time
elapsed, the time consumed by system overhead, and the time used to execute utility to the standard
error stream. Times are reported in seconds.
...

Section 2: System Calls

You’ll probably not use this section unless you are doing systems programming². This section contains info on the avaiable Linux Kernel system calls. For example, running man 2 chown gives:

CHOWN(2) BSD System Calls Manual CHOWN(2)
NAME
chown, fchown, lchown, fchownat -- change owner and group of a file
SYNOPSIS
#include <unistd.h>
int
chown(const char *path, uid_t owner, gid_t group);
...

Section 3: Library Calls

These are the manpages for the C standard library functions. For example, man 3 time:

TIME(3) BSD Library Functions Manual TIME(3)
NAME
time -- get time of day
LIBRARY
Standard C Library (libc, -lc)
SYNOPSIS
#include <time.h>
time_t
time(time_t *tloc);
...

Here we can see why the sections are important to know about.

Running man time would not open the page above, because man searches the library in ascending section order, meaning that it actually finds time(1) and shows the pages for the time program, not the time C library call.

Because of the potential ambiguity of names if no section number is included, in lots of Linux documentation you’ll see the man section number written next to library calls, system calls, programs and so on (things will refer to sed(1) or time(3) for example.

Section 4: Devices

This section deals with the special devices which live in the /dev/* folder. For example, running man 4 random shows:

RANDOM(4) BSD Kernel Interfaces Manual RANDOM(4)
NAME
random , urandom -- random data source devices.
SYNOPSIS
pseudo-device random
DESCRIPTION
The random device produces uniformly distributed random byte values of
potentially high quality.
...

Again, we see that section numbers can be important. If you just run man random, you’ll see:

RANDOM(3) BSD Library Functions Manual RANDOM(3)
NAME
initstate, random, setstate, srandom, srandomdev -- better random num-
ber generator; routines for changing generators
LIBRARY
Standard C Library (libc, -lc)
SYNOPSIS
#include <stdlib.h>
char *
initstate(unsigned seed, char *state, size_t size);
long
random(void);
...

Which is the manpage for random(3), which is C library function, not the /dev/random file!

Section 5: File Formats

This section details special files in the system. For example, man 5 crontab shows:

CRONTAB(5) BSD File Formats Manual CRONTAB(5)
NAME
crontab -- tables for driving cron
DESCRIPTION
A crontab file contains instructions to the cron(8) daemon of the gen-
eral form: ``run this command at this time on this date''. Each user
has their own crontab, and commands in any given crontab will be exe-
cuted as the user who owns the crontab. Uucp and News will usually
have their own crontabs, eliminating the need for explicitly running
su(1) as part of a cron command.
...

Which describes the crontab file used to define scheduled tasks. Again, this is different to man crontab which would document crontab(1). Similarly, man 5 passwd is going to show something quite different to man passwd.

Section 6: Games

Nothing says it better than man 6 intro itself (this’ll not work on a Mac sadly, but try it on another Linux system):

...
DESCRIPTION
Section 6 of the manual describes all the games and funny little programs available on the system.
...

There are probably a few silly programs available on your system, here you’ll find their manuals. For example, man 6 banner on a Mac shows:

BANNER(6) BSD Games Manual BANNER(6)
NAME
banner -- print large banner on printer
SYNOPSIS
banner [-d] [-t] [-w width] message ...
DESCRIPTION
Banner prints a large, high quality banner on the standard output. If
the message is omitted, it prompts for and reads one line of its stan-
dard input.
...

This section is going to be highly dependent on your OS!

Section 7: Miscellaneous

This is where you’ll find additional assorted documentation. For example, man 7 ascii shows:

ASCII(7) BSD Miscellaneous Information Manual ASCII(7)
NAME
ascii -- octal, hexadecimal and decimal ASCII character sets
DESCRIPTION
The octal set:
000 nul 001 soh 002 stx 003 etx 004 eot 005 enq 006 ack 007 bel
...

Section 8: System Commands

We’ve actually already seen one of these commands mentioned, in the manpage for crontab(5) it mentions cron(8). Let’s see, with man 8 cron:

CRON(8) BSD System Manager's Manual CRON(8)
NAME
cron -- daemon to execute scheduled commands (Vixie Cron)
SYNOPSIS
cron [-s] [-o] [-x debugflag[,...]]

These are commands which sysadmins would normally run. You might open section eight unexpectedly, for example man chmod will open chmod(1), but man chown will open chown(8), as it is a system command.

Some distributions might vary for Section Nine. On my Mac it contains information about the kernel interfaces, a C style guide and some more.

Getting the Index of Manual Section

Manpages are just files on the filesystem, so you can get the index of a section just by looking in the appropriate folder.

For example, to index the available system calls, try ls /usr/share/man/man2:

EV_SET.2
FD_CLR.2
FD_COPY.2
FD_ISSET.2
FD_SET.2
FD_ZERO.2
_exit.2
accept.2
access.2
acct.2
...

This is quick and easy way to see what sort of entries you have on your system. If you want to work out where an entry lives, use the -w flag:

$ man -w printf
/usr/share/man/man1/printf.1

Searching the Manual

You can search the manpage titles and summaries with man -k. For example, man -k cpu shows:

cpuwalk.d(1m) - Measure which CPUs a process runs on. Uses DTrace
dispqlen.d(1m) - dispatcher queue length by CPU. Uses DTrace
gasm(n), grammar::me::cpu::gasm(n) - ME assembler

You can find more advanced options for searching by using your newfound man skills on man itself.

That’s Enough!

I’d recommend tldr as a first-call for checking to see how to use a command.

man is a powerful tool to dive deeper into how programs and components of the system work. Like many tools which have been around for a long time, there’s a lot you can do with man. Much of it you’ll likely never need, so I’ve tried to keep this article to the basics.

Understanding manpage sections is useful - you’ll see them referenced again and again in documentation on the system and online.

I hope this helps you save some time when you are working! Please let me know in the comments if you have any questions or thoughts.

You can also check out the rest of the effective shell series.

Appendix: Dash

As a final note, if you find yourself using man a lot because you work offline (I fly a lot so find it very helpful when on a plane with no WiFi), you should also look at Dash³.

Dash is simply an offline documentation aggregator. It can download online manuals for many, many different programming languages, frameworks, technologies and so on. I actually have a vim keyboard command to open the word under the cursor in dash, with the documentation automatically set based on the type of the file.

This is super-useful if you are offline at lot and need more sophisticated offline documentation. You can find out more about it at https://kapeli.com/dash.

Footnotes

Weirdly satisfying to run. ↩︎
Which it is always fun to try if you get the chance, and a great way to learn more about the fundamentals of the operating system. ↩︎
Dash is a paid product. Full disclosure - I don’t get any money from them or anyone else to write about anything, all content is 100% based on my experiences. I don’t run ads on my site either. ↩︎

Effective Shell Part 2: Become a Clipboard Gymnast

Tue, 10 Oct 2017 09:57:54 +0000

This is the second part of my Effective Shell series, which contains practical tips for using the shell to help with every day tasks and be more efficient:

In this article I’ll show you how you can use the shell as an efficient tool to compliment how you use the clipboard.

Note for Linux Users: In this article I’ll use the pbcopy and pbpaste commands to access the clipboard, which are available on a Mac only. To get access to the same commands on other platforms, check Appendix: Clipboard Access on Linux.

Use the Shell on the Clipboard

You can easily use shell commands on the contents of your clipboard. Just use pbpaste to output the clipboard, run the output through some commands, then use pbcopy to copy the result.

Try copying the following text:

Kirk Van Houten
Timothy Lovejoy
Artie Ziff

Then in the shell, run:

pbpaste

You should see the contents of the clipboard. Now we’ll look at some ways that shell access to the clipboard can help with common tasks.

Removing Formatting

Don’t you hate it when you have to copy formatted text and don’t have an easy way to paste it as unformatted text? Here’s an example, I want to copy this Wikipedia page on ‘bash’, and paste it into a Word document:

Many programs have a shortcut to paste the contents of the clipboard (such as ‘command + shift + v’) but if you are like me you might find yourself pasting into a plain text editor just to copy out the plain text.

If you just run the command pbpaste | pbcopy, you can easily strip the formatting:

We’re just piping out the clipboard (which ends up as plain text, cause we’re in a terminal!) and then piping that plain text back into the clipboard, replacing the formatted text which was there before.

This little trick can be very useful. But we can use the same pattern to quickly manipulate the contents of the clipboard in more sophisticated ways.

Manipulating Text

Let’s say someone has emailed me a list of people I need to invite to an event:

The problem is:

The list is in Excel and is formatted
The list has duplicates
I need to turn each name into an email address like ‘Artie_Ziff@simpsons.com’

And I want to email everyone quickly.

We can quickly handle this task without leaving the shell.

Copy the raw text below if you want to try out the same commands and follow along:

Artie Ziff
Kirk Van Houten
Timothy Lovejoy
Artie Ziff
Nick Riviera
Seymore Skinner
Hank Scorpio
Timothy Lovejoy
John Frink
Cletus Spuckler
Ruth Powers
Artie Ziff
Agnes Skinner
Helen Lovejoy

First, we copy the text to the clipboard.

Now we can paste and sort:

$ pbpaste | sort
Agnes Skinner
Artie Ziff
Artie Ziff
Artie Ziff
Cletus Spuckler
Hank Scorpio
Helen Lovejoy
John Frink
Kirk Van Houten
Nick Riviera
Ruth Powers
Seymore Skinner
Timothy Lovejoy
Timothy Lovejoy

Then remove the duplicates:

$ pbpaste | sort | uniq
Agnes Skinner
Artie Ziff
Cletus Spuckler
Hank Scorpio
Helen Lovejoy
John Frink
Kirk Van Houten
Nick Riviera
Ruth Powers
Seymore Skinner
Timothy Lovejoy

Replace the underscore with an ampersand:

$ pbpaste | sort | uniq | tr " " "_"
Agnes_Skinner
Artie_Ziff
Cletus_Spuckler
Hank_Scorpio
Helen_Lovejoy
John_Frink
Kirk_Van_Houten
Nick_Riviera
Ruth_Powers
Seymore_Skinner
Timothy_Lovejoy

Then add the final part of the email address:

$ pbpaste | sort | uniq | tr " " "_" | sed 's/$/@simpsons.com/'
Agnes_Skinner@simpsons.com
Artie_Ziff@simpsons.com
Cletus_Spuckler@simpsons.com
Hank_Scorpio@simpsons.com
Helen_Lovejoy@simpsons.com
John_Frink@simpsons.com
Kirk_Van_Houten@simpsons.com
Nick_Riviera@simpsons.com
Ruth_Powers@simpsons.com
Seymore_Skinner@simpsons.com
Timothy_Lovejoy@simpsons.com

This looks perfect! We can now put the transformed text back onto the clipboard:

$ pbpaste | sort | uniq | tr ' ' '_' | sed 's/$/@simpsons.com' | pbcopy

All in all we have the following pipeline:

pbpaste - output the clipboard
sort - order the output
uniq - deduplicate the rows
tr ' ' '_' - replace spaces with underscores
sed /$/@simpsons.com - add the email domain to the end of the row

Building this in one go is hard, let’s look at little more at the pipeline.

Thinking in Pipelines

Some of these commands might be unfamiliar, some might not make sense, and you might be thinking ‘how would I remember that’. Actually, there are many ways to solve the problem above, this is the one I came up with by iteratively changing my input text.

Here’s what I mean - you’ll see that I actually build a pipeline like this step-by-step:

You can see in the screenshots that I start simple, and step by step add the stages we need.

(P.S - if you are wondering how I am jumping backwards and forwards a word at a time, check the last chapter ‘Navigating the Command Line’).

What we’re doing here is only possible because these simple commands all follow ’the Unix Philosophy’. They do one thing well, and each command expects it’s input to become the input of another command later on. Specifically:

The commands are primitive and simple - sort is sorting a list, uniq is making elements unique.
The commands don’t produce unnecessary output - sort doesn’t add a header such as Sorted Items, which is great because otherwise it would clutter our pipeline.
We are chaining commands together, the output of one becomes the input of another.

We don’t need a command such as ‘Take a muddy list, sort and clean it, then turn pairs of words into an email address’ - with a few simple ‘workhorse’ commands we can easily build this functionality ourselves.

These workhorse commands will be introduced and detailed as we go through the series. We’ll also spend a lot more time looking at pipelines.

I hope this was useful! Please comment if you have any questions or tips. To see further articles as they come out, follow the repo at:

github.com/dwmkerr/effective-shell

Or just follow @dwmkerr on Twitter.

Appendix - Clipboard Access on Linux

If you are using Linux, there is no pbcopy and pbpaste commands. You can use the xclip tool to create equivalent commands.

First, install xclip:

sudo apt-get install -y xclip

Then add the following to your .bashrc file:

# Create mac style aliases for clipboard access.
alias pbcopy="xclip -selection c"
alias pbpaste="xclip -selection c -o"

Obviously you can use any alias you like! The article assumes that pbcopy and pbpaste have been used.

Effective Shell Part 1: Navigating the Command Line

Sun, 11 Jun 2017 23:05:40 +0000

This is the first part of a series I am writing which contains practical tips for using the shell more effectively.

I can’t think of a better place to start than navigating the command line. As you start to do more and more in the shell, text in the command line can quickly become hard to handle. In this article I’ll show some simple tricks for working with the command line more effectively.

Here’s a quick reference diagram, the rest of the article goes into the details!

This article, examples and diagrams are available at github.com/dwmkerr/effective-shell.

Basic Navigation
Searching
Editing In-Place
Clear the Screen
Pro Tip: All The Keys!
Pro Tip: Transposing!
Closing Thoughts

Let’s assume we have a very simple command we are writing, which is going to write a quote to a text file:

echo "The trouble with writing fiction is that it has to make sense,
whereas real life doesn't. -- Iain M. Banks" >> quote.txt

Navigating around long lines of text is a slow process if you are only relying on the arrow keys, so take the time to learn the following shortcuts:

Action	Shortcut	Example
Go to beginning / end	`Ctrl + a`, `Ctrl + e`
Go backwards / forwards one word	`Alt + b` / `Alt + f`
Delete a word / undo	`Ctrl + w` / `Ctrl + -`
Delete next word	`Alt + d`
Delete all the way to the beginning[^1]	`Ctrl + u`
Delete all the way to the end	`Ctrl + k`

Note that if you are on a Mac, you might need to tweak your console to allow the ‘Alt’ key to work.

For iTerm2, go to settings (Command + ,) > Profiles Tab > select the profile you are using > Keys tab. There, you should see Left Option key and Right Option Key with three radio buttons. Select “Esc+” for the Left Option Key.

For Terminal, go to Profiles Tab > Keyboard Tab > check “Use Option as Meta key” at the bottom of the screen.

Searching

Once you have the basic navigation commands down, the next essential is searching. Let’s assume we’ve run the following three commands:

$ command1 param1 param2 param3
$ command2 param4 param5 param6
$ command3 param7 param8 param9

You can search backwards or forwards with Ctrl + r and Ctrl + s. This will search in the current command and then iteratively through previous commands:

This is useful for searching in the current command, but can be also used to quickly search backwards and forwards through the command history:

As you type, your command history is searched, the most recent commands coming first. Use the arrow keys to edit the command, press enter to execute it, or Ctrl + g to cancel the search.

Here are the same commands applied to the original example:

Action	Shortcut	Example
Search backwards / forwards	`Ctrl + r` / Ctrl + s
Run the command	`Enter`
Edit the command	`Right Arrow` / `Right Arrow`
Stop searching	`Ctrl + g`

Editing In-Place

These tips and tricks are helpful, but if you are working with a really long or complex command, you might find it useful just to jump into your favourite editor.

Use Ctrl + x , Ctrl + e to edit-in place:

In a later article I’ll talk a little more about how to configure the default editor.

Clear the Screen

Probably the shortcut I use the most is Ctrl + l, which clears the screen without trashing your current command. Here’s how it looks:

Pro Tip: All The Keys!

You can use the bindkey command to see a list of all keyboard shortcuts:

$ bindkey
"^@" set-mark-command
"^A" beginning-of-line
"^B" backward-char
"^D" delete-char-or-list
"^E" end-of-line
"^F" forward-char
"^G" send-break
"^H" backward-delete-char
"^I" expand-or-complete
"^J" accept-line
"^K" kill-line
"^L" clear-screen
...

This is an extremely useful command to use if you forget the specific keyboard shortcuts, or just want to see the shortcuts which are available.

Pro Tip: Transposing!

If you’ve mastered all of the commands here and feel like adding something else to your repertoire, try this:

The Alt + t shortcut will transpose the last two words. Use Ctrl + t to transpose the last two letters:

These were new to me when I was researching for this article. I can’t see myself ever being able to remember the commands more quickly than just deleting the last two words or characters and re-typing them, but there you go!

Closing Thoughts

If you are ever looking to go deeper, then search the web for GNU Readline, which is the library used under the hood to handle the command line in many shells. You can actually configure lower level details of how all shells which use readline work, with the .inputrc configuration file.

The great thing about learning these shortcuts is that they will work in any prompt which uses GNU Readline. This means everything you’ve learnt applies to:

Bash
zsh
The Python REPL
The Node.js REPL

And probably a whole bunch more¹.

All of these shortcuts should be familiar to Emacs users. There is in fact a ‘Vi Mode’ option for readline, which allows you to use vi commands to work with text. You can enter this mode with set -o vi, I’ll likely come back to this in detail in a later article.

There’s a great cheat sheet on emacs readline commands at readline.kablamo.org/emacs, which is a very useful reference if you want to dig deeper. For this article I’ve tried to focus on what I think are the most useful commands (and transpose just so you can show off!).

Hope that was useful! GIFs were made with LICEcap.

Footnotes

References

If you know of any more, please let me know and I’ll update the article! ↩︎

Simple Continuous Integration for Docker Images

Thu, 03 Nov 2016 05:14:35 +0000

In this article I’m going to demonstrate a few tips and tricks which can make your life easier when you are building or maintaining Dockerfiles.

The need for a Build Pipeline

Do we really need any kind of continuous integration or build pipeline for Dockerfiles?

There will be cases when the answer is no. However, if the answer to any of the following questions is ‘yes’, it might be worth considering:

Do you want others to be able to contribute to the Dockerfile, perhaps changing the image over time?
Are there specific functionalities in your Dockerfiles which could break if altered?
Do you expect to need to release updates to your Dockerfile?

Essentially, if we are looking at providing some kind of automated quality assurance and automation around building and releasing, then a build pipeline is not a bad idea.

A simple Build Pipeline

Here’s what a simple build pipeline could look like. This example is for a Docker Image I just created for local DynamoDB development - dwmkerr/docker-dynamodb:

Let’s dissect what we’ve got here.

The Dockerfile

This is the main ‘code’ of the project if you like. The Dockerfile is the recipe for the image we create.

The Continuous Integration Service

In this case, I am using CircleCI, however the approach described would work fine with most CI systems (such as Jenkins, TravisCI and TeamCity). There is an option to use the Docker Hub Automated Builds, but I’ve found this doesn’t give the flexibility I need (see Why not Docker Hub Automated Builds).

Essentially the CI service needs to offer the option to have three distinct steps in the pipeline, each of which must pass for process to proceed:

Build
Test
Deploy

The Build

We can build with tools, script files, whatever. At the moment, I am leaning towards makefiles. Normally I only need a few lines of shell script to do a build - anything more complex and the makefile can call a shell script. See also Why Makefiles?

Here’s what it might look like:

build:
docker build -t dwmkerr/dynamodb:latest .
ifndef BUILD_NUM
$(warning No build number is defined, skipping build number tag.)
else
docker build -t dwmkerr/dynamodb:$(BUILD_NUM) .
endif

This command just builds the Dockerfile and tags it as dwmkerr/dynamodb:lastest. If a BUILD_NUM variable is present, we also create the tag dwmkerr/dynamodb:BUILD_NUM. This means if we want to deploy to a service such as Amazon ECS we can push a specific build by referring to the image with that tag.

The Tests

Again I’m relying on make. I just want to be able to run make test - if zero is returned I’m happy. If not, the pipeline should stop and I’ll check the output. Here’s my test command:

test: build
./test/basics.test.sh
./test/ephemeral.test.sh
./test/persistent.test.sh

Not a thing of beauty, but it works. These scripts I’ll discuss a little bit later on, in the delightly titled What are these test scripts section.

For CircleCI, this is enough to have the main part of our pipeline. Here’s how the circle.yml file looks at this stage:

machine:
services:
- docker
environment:
# Set the build number, used in makefiles.
BUILD_NUM: $CIRCLE_BUILD_NUM
test:
override:
- make test

(Actually there’s a couple of other bits but they’re just to make sure circle uses the right version of Docker, see the full circle.yml file here).

The Deployments

Deployments are trivial as all we need to do is push to the Docker Hub. The make deploy command looks-a like this:

deploy:
docker push dwmkerr/dynamodb:latest
ifndef BUILD_NUM
$(warning No build number is defined, skipping push of build number tag.)
else
docker push dwmkerr/dynamodb:$(BUILD_NUM)
endif

We’re pushing the latest tag and BUILD_NUM tag if present. To add this to the CircleCI pipeline, we just add the following to circle.yml:

deployment:
master:
branch: master
commands:
- docker login -e $DOCKER_EMAIL -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
- make deploy

If we have a push to master, we log in to Docker (using environment variables I configure in the CircleCI UI) and then run make deploy to push our images.

That’s It

That’s about it. This is a pretty simple approach, you can see it in action at:

github.com/dwmkerr/docker-dynamodb

The rest of this post is a bit of a deep dive into some specific areas I found interesting.

Appendix 1: Why not Docker Hub Automated Builds?

There are automated builds available in the Docker Hub:

I’m not using this feauture at the moment, here’s a brief roundup of what I think are the current pros and cons:

Pros

You don’t have to goof around installing Docker on a CI platform.
It allows you to update the description of your Docker image automatically, from the GitHub README.md.
It allows you to associate the image with a specific GitHub repo (rather than just linking from the image description).
Branch management - allowing tags to be built for specific branches.

Cons

It doesn’t seem to support any kind of configurable gating, such as a running a test command prior to deploying.
It doesn’t seem to support any kind of triggering of downstream processes, such as updating environments, sending notifications or whatever.

The lack of ability to perform tests on the image before deploying it why I’m currently not using the service.

By doing the testing in a CI system for every pull request and only merging PRs where the tests pass we could mitigate the risk here. This service is worth watching as I’m sure it will evolve quickly.

Appendix 2: Why Makefiles?

I started coding with a commandline compiler in DOS. When I used my first GUI (Borland Turbo C++) it felt like a huge leap:

Later on I moved onto Microsoft Visual C++ 4.2:

And you cannot imagine the excitement when I got my boxed edition of Visual Studio .NET:

Wow!

Anyway, I digress. GNU make was invented by Leonardo Da Vinci in 1473 to allow you to build something from the commandline, using a fairly consistent syntax.

It is near ubiquitous on *nix systems. I am increasingly using it as an ’entry point’ to builds, as I use variety of languages and platforms. Being able to know that most of the time:

make build
make test

Will build and test something is convenient. Makefiles actually are not that great to work with (see this, this and this). I’ve found as long as you keep the commands simple, they’re OK. For anything really complex, I normally have a scripts/ folder, but call the scripts from the makefile, so that there’s still a simple entrypoint.

I’m not entirely sold on makefiles, but they tend to be my default at the moment if I know I’m going to use the commandline for builds (for example, in Java projects I’ll often write a makefile to call Maven or Gradle).

For things like Node.js, where you have commands like npm test or npm run xyz I still sometimes use makefiles, using npm for day-to-day dev tests (npm start) and make if it’s something more complex (e.g. make deploy-sit to deploy to an SIT environment).

Appendix 3: What are these test scripts?

You may have noticed:

test: build
./test/basics.test.sh
./test/ephemeral.test.sh
./test/persistent.test.sh

What’s going on here?

My Docker image is just a wrapper around Amazon’s Local DynamoDB tool. I don’t really need to test that tool. But what I wanted to test was the capabilities which lie at the intersection between ’native’ Docker and ’native’ DynamoDB.

For example, I know Docker supports volume mapping. I know DynamoDB supports using a data directory, to allow persistent between runs. I want to test I can combine Docker volume mapping and the DynamoDB data directory features. I know Docker images should default to being ephemeral, I want to test this holds true by default for my image.

Testing Docker is a little hard - I want to test that I can run containers, start, stop, check state before and after and so on. This is essentially an integration test, it can be tricky to make it truly isolated and deterministic.

I’ve given it my best go with these scripts. Here’s an example for the ’ephemeral’ test, where I’m trying to assert that if I run a container, create a table, stop the container and run a new one, I no longer have the table. Here’s the test:

# Bomb if anything fails.
set -e

# Kill any running dynamodb containers.
echo "Cleaning up old containers..."
docker ps -a | grep dwmkerr/dynamodb | awk '{print $1}' | xargs docker rm -f || true

# Run the container.
echo "Checking we can run the container..."
ID=$(docker run -d -p 8000:8000 dwmkerr/dynamodb)
sleep 2

# Create a table.
aws dynamodb --endpoint-url http://localhost:8000 --region us-east-1 \
 create-table \
 --table-name Supervillains \
 --attribute-definitions AttributeName=name,AttributeType=S \
 --key-schema AttributeName=name,KeyType=HASH \
 --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

# Clean up the container. On CircleCI the FS is BTRFS, so this might fail...
echo "Stopping and restarting..."
docker stop $ID && docker rm $ID || true
ID=$(docker run -d -p 8000:8000 dwmkerr/dynamodb)
sleep 2

# List the tables - there shouldn't be any!
COUNT=$(aws dynamodb --endpoint-url http://localhost:8000 --region us-east-1 \
 list-tables \
 | jq '.TableNames | length')
if [ $COUNT -ne "0" ]; then
 echo "Expected to find no tables, found $COUNT..."
 exit 1
fi

It’s a bit dirty - it removes containers from the host, changes things and so on. But it works.

I did experiment with running these tests in a container, which has the benefit of giving you a clean host to start with, which you can throw away after each test.

I had to give up after a little while due to time constraints, but will probably revisit this process. The benefits of running these integration tests in a container is that we get a degree of isolation from the host.

If anyone is interested, my attempts so far are on this RFC Pull Request. Feel free to jump in!

Quick Tip: Sending Newlines with cURL

Tue, 03 May 2016 22:12:28 +0000

Yikes, this took far too long to figure out!

I have a service which takes plain text multi-line input and outputs an object for each line, something like this:

Input

Line 1
Line 2
Line 3

Output

[
{line: "Line 1"},
{line: "Line 2"},
{line: "Line 3"}
]

There’s a bit more to it than that, but that’s the gist.

I want to test my service with cURL, trying:

curl --data "Line 1\nLine 2\nLine 3" \
-H "Content-Type: text/plain" localhost:3000/parse

This did not work. Nor did some alternatives. And I really didn’t want to have to write the text to a file and load it in.

Turns out there’s a nice little shell trick to let you use escape characters C style, use $'some\ncontent' to use ANSI C escaping. Now you can cURL with newlines!

curl --data $'Line 1\nLine 2\nLine 3' \
-H "Content-Type: text/plain" localhost:3000/parse

Enjoy!

Bash on dwmkerr.com

A simple Makefile 'help' command

How it Works

Testing Scripts

Further Reading

Effective Shell for Beginners

Effective Shell Part 7: The Subtleties of Shell Commands

What Are Commands?

Executables - Programs

Executables - Scripts

Builtins

Functions

Aliases

So What?

More than You Need to Know

what

whatis

which

whence

where

whereis

command

type

Effective Shell Part 6: Everything You Don't Need To Know About Job Control

What Is Job Control?

The Problem

Solution 1: Start the Server in the Background

Solution 2: Move the Server to the Background

Moving Background Jobs to the Foreground

Cleaning Up Jobs

Why You Shouldn’t Use Jobs

How to Escape Jobs

Alternatives to Jobs

Quick Reference

Footnotes

Effective Shell Interlude: Understanding the Shell

Introduction for the Non-Technical Reader

Introduction for the Technical Reader

Let’s Get Started!

A Computer in a Nutshell

The Operating System

The Kernel

User Space

The Shell

The Terminal

Back to the Shell

The Command Prompt or Command Line

Shell Commands and Different Shells

That’s a Wrap!

Example: iTerm 2 / tmux / zsh

Example: Bash

Example: Windows Explorer

Example: Windows Command Prompt

Example: Windows PowerShell

Example: Windows Subsystem for Linux (WSL)

Share and Discuss

Effective Shell 4: Move Around!

Getting Comfortable Moving Around

Where Am I?

Changing Directory

Exploring a Directory

Managing the Directory Stack

Auto-Completion

That’s It!

A portable and magic-free way to open Pull Requests from the Command Line

The Script

How It Works

Prior Art

Effective Shell Part 3: Getting Help

Getting Help is Important!

tl;dr

Understanding ‘man’

Getting help on a command

Using the pager

What’s with the numbers?

Section 1: Programs and Shell Commands

Section 2: System Calls

Section 3: Library Calls

Section 4: Devices

Section 5: File Formats

`what`

`whatis`

`which`

`whence`

`where`

`whereis`

`command`

`type`