SysAdmin on dwmkerr.com

Effective Shell Part 7: The Subtleties of Shell Commands

Tue, 25 Jun 2019 07:25:23 +0000

In this chapter, we’ll take a look at the various different types of shell commands that exist and how this can affect your work.

By the end of this chapter, you might even be able to make sense of the horrifying and perfectly syntactically valid code below:

which $(where $(what $(whence $(whereis who))))

What Are Commands?

This is really important to understand! A command in a shell is something you execute. It might take parameters. Generally it’ll have a form like this:

command param1 param2

We’ve already seen many commands during this series:

ls # Show the contents of the current directory
cd ~ # Move to the user's home
cat file.txt # Output the contents of 'file.txt' to stdout

But to be an effective shell user, you must understand that not all commands are created equal. The differences between the types of commands will affect how you use them.

There are four types of commands in most shells:

Executables
“Built-Ins” (which we’ll just call builtins from now on)
Functions
Aliases

Let’s quickly dig in and see a bit more.

Executables - Programs

Executables are just files with the ’executable’ bit set¹. If I execute the cat command, the shell will search for an executable named cat in my $PATH. If it finds it, it will run the program.

$ cat file.txt
This is a simple text file

What is $PATH? $PATH is the standard environment variable used to define where the shell should search for programs. If we temporarily empty this variable, the shell won’t find the command:

$ PATH="" cat file.txt
bash: cat: No such file or directory

Normally your $PATH variable will include the standard locations for Linux programs - folders such as /bin, /sbin, /usr/bin and so on².

If you were to print the variable, you’d see a bunch of paths (they are separated by colons; I’ve put them on separate lines for readability):

/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin

The shell will start with the earlier locations and move to the later ones. This allows local flavours of tools to be installed for users, which will take precedence over general versions of tools.

There will likely be other locations too - you might see Java folders, package manager folders and so on.

Executables - Scripts

Imagine we create a text file called dog in the local folder:

#!/bin/sh
echo "🐶 woof 🐶"

If we make the file executable, by running chmod +x dog³, then we can run this just like any other program - as long as we tell the shell to look for programs in the current directory:

$ PATH="." dog
🐶 woof 🐶

More common would be to run the program by giving a path:

$ ./dog
🐶 woof 🐶

Or just move it to a standard location that the shell already checks for programs:

$ mv dog /usr/local/bin
$ dog
🐶 woof 🐶

The point is that executables don’t have to be compiled program code. If a file starts with #! (the ‘shebang’), then the system will try to run the contents of the file with the program specified in the shebang.

We will look at shebangs in greater detail in a later chapter.

Builtins

OK, so we’ve seen executables. What about a command like this?

local V="hello" echo $V

You will not find the local executable anywhere on your system. It is a builtin - a special command built directly into the shell program.

Builtins are often highly specific to your shell. They might be used for programming (local for example is used to declare a locally scoped variable), or they might be for very shell-specific features.

This is where we need to take note. As soon as you are running a builtin, you are potentially using a feature that is specific to your shell, rather than a program that is shared across the system and can be run by any shell.

Trying to programmatically execute local as a process will fail - there is no executable with that name; it is purely a shell construct.

So how do we know if a command is a builtin? The preferred method is to use the type command:

$ type local
local is a shell builtin

The type command (which is itself a builtin!) can tell you the exact type of shell command.

Interestingly, you might be using more builtins than you think. echo is a program, but most of the time you are not executing it when you are in a shell:

$ type -a echo
echo is a shell builtin
echo is /bin/echo

By using the -a flag on type to show all commands that match the name, we see that echo is actually both a builtin and a program.

Many simple programs have builtin versions. The shell can execute them much faster.

Some commands are a builtin so that they can function in a sensible manner. The cd command changes the current directory - if we executed it as a process, it would change only the directory for the cd process itself, not the shell, making it much less useful.

Builtins will vary from shell to shell, but many shells are ‘Bash-like’ - meaning they will have a set very similar to the Bash builtins, which you can see here:

https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html

As should be familiar from Part 3: Getting Help, you can get help for builtins:

$ man source # source is a builtin
BUILTIN(1) BSD General Commands Manual BUILTIN(1)

NAME
 builtin, !, %, # ...snip...

SYNOPSIS
 builtin [-options] [args ...]

However, the manual will not show information on specific builtins, which is a pain. Your shell might have an option to show more details - for example, in Bash you can use help:

$ help source
source: source filename [arguments]
 Read and execute commands from FILENAME and return. The pathnames
 in $PATH are used to find the directory containing FILENAME. If any
 ARGUMENTS are supplied, they become the positional parameters when
 FILENAME is executed.

But remember: help is a builtin; you might not find it in all shells (you won’t find it in zsh, for example). This highlights again the challenges of builtins.

Functions

You can define your own shell functions. We will see a lot more of this later, but let’s show a quick example for now:

$ restart-shell () { exec -l "$SHELL" }

This snippet creates a function that restarts the shell (quite useful if you are messing with shell configuration files or think you might have irreversibly goofed up your current session).

We can execute this function just like any command:

$ restart-shell

And running type will show us that this is a function:

$ type restart-shell
restart-shell is a function
restart-shell ()
{
 exec -l $SHELL
}

Functions are one of the most powerful shell constructs we will see; they are extremely useful for building sophisticated logic. We’re going to see them in a lot more detail later, but for now it is enough to know that they exist, and can run logic, and are run as commands.

Aliases

An alias is just a shortcut. Type in a certain set of characters, and the shell will replace them with the value defined in the alias.

Some common commands are actually already aliases - for example, in my zsh shell, the ls command is an alias:

% type -a ls
ls is an alias for ls -G
ls is /bin/ls

I make sure that when I use the ls command, the shell always expands it to ls -G, which colours the output.

We can quickly define aliases to save on keystrokes. For example:

$ alias k='kubectl'

From this point on, I can use the k alias as shorthand for the kubectl command.

Aliases are far less sophisticated than functions. Think of them as keystroke savers and nothing more, and you won’t go far wrong. Aliases are not portable across shells and have certain behaviours which can make them problematic to work with, there will be an entire chapter dedicated to alisases coming up in the series.

So What?

So we now hopefully have a greater understanding of the variety of shell commands. Not all commands are executables, not all of the commands we think are executables necessarily are, and some commands might be more sophisticated.

As a shell user, the key things to remember are:

Executables are ‘safe’ - they are programs your system can use; your shell just calls out to them.
Builtins are very shell-specific and usually control the shell itself
Functions are powerful ways to write logic but will normally be shell-specific.
Aliases are conveniences for human operators, but only in the context of an interactive shell.

To find out how a command is implemented, just use the type -a command:

$ type -a cat
cat is /bin/cat

More than You Need to Know

OK, for the masochistic few, you might be wondering about all of the other commands and utilities you may have seen that can tell you about programs and commands:

what
whatis
which
whence
where
whereis
command
type

A lot of these are legacy and should be avoided, but for completeness sake, we’ll go through them.

`what`

what reads out special metadata embedded in a program, generally used to identify the version of source code it was built from:

$ what /bin/ls
/bin/ls
 Copyright (c) 1989, 1993, 1994
 PROGRAM:ls PROJECT:file_cmds-272.220.1

There should be almost no circumstance in which you need to use it in your day-to-day work, but you might come across it if you meant to type whatis.

`whatis`

whatis searches a local help database for text. This can be useful in tracking down manual pages:

$ whatis bash
bash(1) - GNU Bourne-Again SHell
bashbug(1) - report a bug in bash

But I can’t imagine it will be a regularly used tool by most users.

`which`

which will search your $PATH to see whether an executable can be found. With the -a flag, it will show all results.

$ which -a vi
/usr/local/bin/vi
/usr/bin/vi

which originated in csh. It remains on many systems for compatibility but in general should be avoided due to potentially odd behaviour⁴.

`whence`

whence was added to the Korn shell. You are unlikely to use it unless you are on systems using ksh. zsh also has this command, but it should be avoided and considered non-standard.

% whence brew
/usr/local/bin/brew

`where`

This is a shell builtin that can provide information on commands, similar to type:

% where ls
ls: aliased to ls -G
/bin/ls

However, type should be preferred, as it is more standard.

`whereis`

whereis is available on some systems and generally operates the same as which, searching paths for an executable:

% whereis ls
/bin/ls

Again, type should be preferred for compatability.

`command`

command is defined in the POSIX standard, so should be expected to be present on most modern systems. Without arguments, it simply executes a command. With the -v argument, you get a fairly machine-readable or processable response; with the -V argument, you get a more human readable response:

% command -v ls
alias ls='ls -G'
% command -V ls
ls is an alias for ls -G

command can be useful in scripts, as we will see in later chapters.

`type`

type is part of the Unix standard and will be present in most modern systems. As we’ve already seen, it will identify the type of command as well as the location for an executable:

% type -a ls
ls is an alias for ls -G
ls is /bin/ls

This command can also be used to only search for paths:

% type -p ls
ls is /bin/ls

Summary

In summary, avoid anything that starts with ‘w’! These are legacy commands, generally needed only when working on older Unix machines. type or command should be used instead.

Footnotes

We will cover permissions and modes in later chapters. ↩︎
Why these names and locations? It’s a long story. The best place to start if you are intersted is the Filesystem Hierarchy Standard. ↩︎
chmod changes the mode of a file; +x means ‘add the executable bit’. This tells the operating system the file can be executed. ↩︎
Stack Exchange: Why not use “which”? What to use then? ↩︎

Effective Shell Part 6: Everything You Don't Need To Know About Job Control

Mon, 10 Jun 2019 08:26:33 +0000

Job control is a feature of most shells, which is generally not particularly intuitive to work with. However, knowing the basics can help prevent you from getting yourself into a tangle, and can from time to time make certain tasks a little easier.

In this chapter, we’ll look at the main features of job control, why it can be a problematic, and some alternatives.

What Is Job Control?

Let’s start with an example. I am building a simple web page. It has one index.html file, one styles.css file, and one code.js file. The index.html file looks like this:

<html>
 <head>
 <title>My New Project</title>
 <link rel="stylesheet" type="text/css" href="styles.css">
 <script src="code.js"></script>
 </head>
 <body>
 <!-- Snip... -->
 </body>
</html>

Opening the file in a browser doesn’t quite work, as it won’t load the code or the styles. We need a web server to serve styles and code.

A super-useful one-liner to run a web server on any machine with Python installed is:

python -m SimpleHTTPServer 3000

In fact, this is so useful that I normally alias this command, so that I can just type serve. We’ll see aliases in a later chapter.

For now, if we run this command (you can get the three sample files here if you want to try this yourself), then we can open the webpage in a browser, with the styles and code loaded:

We can also see that the server has served the HTML, JavaScript, and CSS files:

All well and good so far.

The Problem

Let’s say we want to now continue using our shell, maybe to edit the website with a terminal editor like Vim or Emacs, or we want to zip up the site, or just run any shell command¹.

We have a problem. The python process is still running - it’s serving the website. Our shell is essentially useless, until we stop the server. See what happens when I try to edit a file:

In the example above, I try to run vi, but nothing is happening. Standard input is not being read by the server and not being interpreted by the shell.

I have to kill the server by hitting Ctrl+C (which sends a SIGINT² - we’ll see more about signals later), clear my screen to get rid of all of the error messages, then start again.

This is obviously not optimal. Let’s look at some solutions.

Solution 1: Start the Server in the Background

In most shells, you can run a command and instruct the shell to run it in the background. To do this, you end the line with an ampersand. Here’s how the example would look in this case:

By ending the command with an & ampersand symbol, we instruct the shell to run the command as a background job. This means that our shell is still functional. The shell has also notified us that this command is running as a background job with a specific job number:

% python -m SimpleHTTPServer 3000 &
[1] 19372

In slightly obtuse language, the shell has informed us that it has started a job in the background, with job number 1 and that this job is currently handling the process with ID 19372.

The ampersand solution is a fairly common pattern used in day-to-day work.

Solution 2: Move the Server to the Background

Let’s say you forgot to start the command in the background. Most likely in this case you’d kill the server with Ctrl+C and then start it again with the & option. However, what if this was a large file download or a task you didn’t want to abort?

In the example below, we’ll move the job to the background:

The process is currently in the foreground, so my shell is inactive. Hitting Ctrl+Z sends a ‘suspend’ signal to the process³, pausing it and moving it to the background.

Let’s dissect this:

% python -m SimpleHTTPServer 3000
Serving HTTP on 0.0.0.0 port 3000 ...
127.0.0.1 - - [03/Jun/2019 13:38:45] "GET / HTTP/1.1" 200 -
^Z
[1] + 21268 suspended python -m SimpleHTTPServer 3000

The shell echos as I type, so we see ^Z (i.e., the Ctrl+Z chord I entered). The shell responds by moving the process into a background job and suspending it.

The key here is that it is suspended. The process is paused. So the web server is no longer serving. If you are following with the sample, reload your browser. The webpage fails to load, as the server process is not able to respond to requests.

To continue the job, in the background, we use the bg (‘background’) command, with a job identifier (which always starts with a % symbol - we’ll see why soon) to tell the shell to continue the job:

% bg %1
[1] + 21268 continued python -m SimpleHTTPServer 3000

The shell lets us know the job is being continued, and if we load the webpage again, the content is shown as expected.

As a final check, we run the jobs command to see what jobs the shell is running:

% jobs
[1] + running python -m SimpleHTTPServer 3000

And there you have it - our server is running as a background job. This is exactly what we would see if we run jobs after starting the server with an & at the end. In fact, using an & is perhaps an easier way to remember how to continue a suspended job:

% %1 &
[1] + 21268 continued python -m SimpleHTTPServer 3000

In the same way ending a command with & runs it in the background, ending a job identifier with & continues it in the background.

There is at least one more way to move a job to the background⁴, but I have not yet found it useful in any scenarios, and it is overly complex to explain. See the footnote for details if you are interested.

Moving Background Jobs to the Foreground

If you have a job in the background, you can bring it back to the foreground with the fg (‘foreground’) command. Let’s show the jobs, with the jobs command:

% jobs
[1] + running python -m SimpleHTTPServer 3000

Here I have a background job running a server. Any one of the following commands will bring it back to the foreground:

fg %1 # Explicitly bring Job 1 into the foreground

%1 # ...or in shorthand, just enter the job id...

fg # ...if not given an id, fg and bg assume the most recent job.

Now the job is in the foreground, and you can interact with the process again however you like.

Cleaning Up Jobs

You might realise you cannot continue what you are doing because an old job is still running. Here’s an example:

I tried to run my web server, but there was still one running as a background job. The server failed to start because the port is in use.

To clean it up, I run the jobs command to list the jobs:

% jobs
[1] + suspended python -m SimpleHTTPServer 3000

There’s my old web server. Note that even though it is suspended, it’ll still be blocking the port it is serving on⁵. The process is paused, but it is still holding onto all of the resources it is using.

Now that I know the job identifier (%1 in this case), I can kill the job:

% kill %1
[1] + 22843 terminated python -m SimpleHTTPServer 3000

This is why job identifiers start with a percentage sign! The kill command I have used is not a special job control command (like bg or fg). It is the normal kill command, which terminates a process. But shells that support job control can normally use a job identifier in place of a process identifier. So rather than working out what the process identifier is that I need to kill, I can just use the job identifier⁶.

Why You Shouldn’t Use Jobs

Avoid jobs. They are not intuitive to interface with, and they suffer from some serious problems.

The most obvious one is that all jobs write to the same output, meaning you can quickly get garbled output like this:

This is what happens when I run a job, which just outputs text every second. It’s in the background, but it’s printing all over my commands. Even running the jobs command to try and find the job to stop it is difficult.

Input is even more complex. If a job is running in the background, but requires input, it will be silently suspended. This can cause confusion.

Jobs can be used in scripts but must be done so with caution and could easily confuse a consumer of the script if they leave background jobs hanging around, which cannot be easily cleaned up⁷.

Handling errors and exit codes for jobs can be problematic, causing confusion, poor error handling, or overly complex code.

How to Escape Jobs

If there are two things to take away, it would be this:

If you have started running a command in the foreground, and you don’t want to stop it and would rather move it to the background, hit Ctrl+Z. Then Google “job control”.

And:

If you think there is a job running in the background, and it is messing with your screen, type fg to bring it to the front and kill it with Ctrl+C. Repeat as needed!

In either case, if you need to do something more subtle, you can return to this reference. But the first command should allow you to get your shell back while you work out how to continue the job, and the second should kill a background job that is messing with your screen.

Alternatives to Jobs

If you are using any kind of modern terminal such as iTerm, Terminal or the GNOME Terminal, just open a new tab or split! Much easier.

The benefit to this is that each tab gets its own standard input and output, so there’s no risk of overwriting. And of course you can hide/reveal/rearrange the tabs however you like.

The traditional alternative to a job for an operator who simply wants more than one thing going on at once would be a terminal multiplexer, such as screen or tmux:

Multiplexers work in a very similar way to a modern graphical terminal - they manage many shell instances. The benefits to a modern terminal, such as iTerm, is that you have a very intuitive GUI and lots of features.

The benefits to a multiplexer are that you can run them over SSH sessions to manage complex operations on remote machines and that they run a client-server model, meaning many people can work with many multiplexed processes (and they can persist beyond sessions).

My personal preference is both - I use a modern terminal and run everything inside it in tmux. We’ll look at both of these options in later chapters.

Quick Reference

You might find that jobs are useful, or you might find that they are not. Either way, here’s a quick reference of some common commands:

Command	Usage
`command &`	Run the command as a background job.
`<Ctrl+Z>`	Move the current process into a background job, suspended.
`jobs`	List all jobs.
`fg %1`	Move background job number 1 into the foreground.
`bg %1`	Continue background job number 1.
`kill %1`	Terminate job number 1.
`wait %1`	Block until job number 1 exits.

If you want to find out more about the gory details of jobs, the best place to start is the Bash Manual - Job Control Section, or the ‘Job Control’ section of your preferred shell’s manual.

I hope you found this useful, and, as always, please leave comments, questions or suggestions below!

Footnotes

If you are not a heavy shell user, this might seem unlikely. But if you do a lot of work in shells, such as sysadmin, devops, or do your coding from a terminal, this happens all the time! ↩︎
Signals like SIGINT, SIGKILL, SIGTERM and so on will be covered in a later chapter. ↩︎
Technically, SIGTSTP - which is ‘TTY stop’. If you have always wondered about the ‘TTY’ acroynm, check the previous chatper, Interlude: Understanding the Shell. ↩︎
The alternative method is to use Ctrl+Y, which will send a delayed interrupt, which will continue to run the process until it tries to read from stdin. At this point, the job is suspended and the control given to the shell. The operator can then use bg or kill or fg to either move to the background, stop the process, or keep in the foreground as preferred. See: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Job-Control ↩︎
Another super-useful snippet: lsof -i -P -n | grep 8000 to find any process that has a given port open. Another one for the aliases chapter! ↩︎
There are times this is needed. If a job runs many processes - for example, by running a pipeline - the process identifier will change as the command moves from one stage of the pipeline to the next. The job identifier will remain constant. Remember, a job is a shell command, so could run many processes. ↩︎
To see how bad this can be, create a script that starts jobs, then run it. Then run the jobs command to see what is running. The output might surprise you! ↩︎