Devops on dwmkerr.com

Semantic Versioning, Conventional Commits and Release Pull Requests for .NET with Google's Release Please Project

Sat, 29 Apr 2023 00:00:00 +0000

Release Please is an excellent library from Google that simplifies the management of releases for projects. However, it is not immediately apparent from the Release Please documentation that you can very easily get this working for your .NET projects.

In this article we’ll take a quick tour of how to configure release please for a .NET project and how it can save you time and effort.

What is Release Please?

In a nutshell, Release Please is a tool that can be run as a CLI or a GitHub action, that looks over the commit history for a repository. Based on the commit history, the tool will attempt to identify whether there have been major, minor or patch level changes to the code, based on the semantic versioning spec and the usage of Conventional Commits. It will then open a pull request that updates the changelog for your project and sets a new version number, and allows you to then easily run custom release actions if the pull request is merged.

This is easier to see with a few screenshots. I recently added Release Please to my SharpGL project. After making a change, I get a pull request like this:

Going into the details of the pull request, what is particularly useful is that my changelog has been updated, as well as the version data in my SharedAssemblyInfo.cs file:

How does this help?

If you maintain a lot of open source projects, anything you can do to automate chores like this just brings rigour to the process and makes bringing in contributions far easier.

A bug had been raised on the repo:

https://github.com/dwmkerr/sharpgl/issues/220

This bug indicated that the published version on NuGet was mis-aligned with the version in the source code. This is obviously a real problem for consumers of the library as well as contributors.

I use Release Please for most of my projects now, I particularly like having the release pull request to review, it makes it really easy to see what’ll be in the next published vesrion.

How do you configure Release Please for .NET?

It turns out that this is really easy! You can configure you manifest like so:

{
 "release-type": "simple",
 "package-name": "sharpgl",
 "extra-files": [
 "SharpGL/AssemblyInfo.cs"
 ]
}

Or directly in your GitHub pipeline configuration:

name: main
on:
 push:
 branches:
 - main
jobs:
 main:
 runs-on: ubuntu-20.04
 steps:
 - uses: google-github-actions/release-please-action@v3
 id: release
 with:
 token: ${{ secrets.GITHUB_TOKEN }}
 release-type: simple
 package-name: sharpgl
 extra-files: |
 SharpGL/AssemblyInfo.cs

It is not immediately apparent from the Release Please documentation that you can manage version numbers in a C# file, but the after looking at the Customising Guide, it turns out there is a ‘generic’ file updater that simply looks for version numbers, based on regexes, in any arbitrary files. You just need to use some comments to show where these version numbers are. In my case, the code looks like this:

using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

[assembly: AssemblyProduct("SharpGL")]
[assembly: AssemblyCopyright("Copyright © Dave Kerr 2023")]

// x-release-please-start-version
[assembly: AssemblyVersion("3.0.0.0")]
[assembly: AssemblyFileVersion("3.0.0.0")]
// x-release-please-end

Adding these x-release-please comments is enough for Release Please to be able to identify the lines that need to be updated.

Other Improvements

There are two immediate improvements that you can make on top of these changes to make it even easier to handle releases.

Commit Linting

The first improvement is to look at something like commitlint to ensure that incoming pull requests have commits that meet the conventional commit spec. For Release Please to be able to properly infer the types of changes from the commit history, it is essential that conventional commits are used.

Remember - even with a hook to check commit messages, you’ll need need to confirm that the author of the changes has structured the commit message appropriately, such as correctly identifying breaking changes.

Automated Publishing

It is possible to update your build pipeline to actually publish your packages when the release pull request is merged. Here’s an example for a .NET project:

# The 'main' workflow is used to create the 'Release Please' pull
# request. This PR is opened when we have a new commit to main.
# See:
# https://github.com/googleapis/release-please
name: main
on:
 push:
 branches:
 - main
jobs:
 main:
 runs-on: windows-2022

 steps:
 - name: Checkout
 uses: actions/checkout@v3

 # Note - this is where you put your build / test steps. See:
 # https://github.com/dwmkerr/sharpgl/blob/main/.github/workflows/main.yaml
 # For an example.

 # Now that we know the build runs, create a release PR if needed.
 - uses: google-github-actions/release-please-action@v3
 id: release
 with:
 token: ${{ secrets.GITHUB_TOKEN }}
 release-type: simple
 package-name: sharpgl
 extra-files: |
 source/SharpGL/SharedAssemblyInfo.cs

 # If we are building a release (i.e. we've merged from release-please) then
 # we can deploy.
 - name: Publish
 run: dotnet nuget push ./source/SharpGL/artifacts/packages/*.nupkg --source 'https://api.nuget.org/v3/index.json' --api-key ${{secrets.NUGET_API_KEY}}
 shell: bash # bash for globs in commmand above...
 if: ${{ steps.release.outputs.release_created }}

Summary

I’d highly recommend this approach for .NET projects. Adopting semver and conventional commits should make managing contributions and your changelog easier. Release Please really is the icing on the cake - by having structured commits with strong semantics you can reduce overhead and effort and increase automation.

I have not included much on enforcing conventional commits in the repository or this article at the moment, some of the best tooling for this uses the Node stack (Husky and commitlint specifically). The benefits of enforcing conventional commits may well be worth the extra effort of setting these tools up, it’s really up to you to decide.

Modernising .NET projects for .NET Core and beyond!

Wed, 01 Jul 2020 00:00:00 +0000

The world of .NET is going through a transformation. The .NET Framework is reaching end of life, .NET Core is an increasingly feature rich and robust platform to develop solutions which target Linux, MacOS, embedded devices, containers and more. There’s also the .NET Standard.

But what does this mean for .NET Framework projects? In this article I’ll describe how to modernise your .NET Framework projects for .NET Core, the .NET Standard and .NET 5, which is planned to be released this year. I’ll also explain the high level differences between the platforms and what the consequences of upgrading are for consumers, developers and maintainers.

The .NET Framework, .NET Core and the Future
The Challenge: Modernisation and Compatibility
The Modernisation Process - Introducing our two Villains
The Key Learnings

The .NET Framework, .NET Core and the Future

There’s a lot which has been written on this topic, but it can still be a little overwhelming to understand just how all of these things fit together.

Here’s a simple visual I’ve created to try and put things into context:

I’m only going to cover the bare essentials - but there are links to further reading on each topic if you want to go deeper. This article is mainly going to be focused on the practicality and consequence of migration and re-targeting.

First, the .NET Framework.

The .NET Framework was created in 2002 as set of unified tools and standards to allow developers on the Microsoft Platform to more quickly build solutions, provide interoperability between languages and more. Read more about the .NET Framework.
The .NET Framework rapidly gained popularity, partly due to the convenience of developing in C# rather than Basic or C/C++. C# provided a more developer friendly language than C or C++ for many use cases, and was heavily inspired by Java. Read more about C#.
With the increase in popularity, the .NET Framework started to have more frequent releases and became a standard part of the Windows operating system, installed out of the box rather than on-demand if needed.
However - the .NET Framework only functioned on Microsoft Windows, which greatly limited its potential uses cases, even as more and more engineers used it for Web, Client Applications and mobile.

Enter .NET Core.

Microsoft signaled a radical switch in their strategy with the appointment of Satya Nadella, becoming increasingly focused on open source, and more importantly, deciding that the Microsoft development toolchain should not force users to use Windows as their execution environment
.NET Core was developed as a lightweight version of the .NET Framework, which could run on multiple platforms - including Linux and MacOS. Read more about .NET Core.
In a short period of time .NET Core became more and more feature rich, providing a lot of capabilities for web developers and front-end application developers.

The challenges of divergence and the .NET Standard.

As .NET Core became more feature rich, the API became closer to the .NET Framework - but they are still fundamentally different runtimes. A binary compiled for the .NET Core does not run on the .NET Framework and vice-versa.
To deal with this issue, Microsoft developed the .NET Standard - a specification of a set of APIs. If a runtime offered these APIs, then solutions built on any runtime which meets the standard could run on any compliant platform.

What does this mean? Basically, the table below shows the consequences of this. If you build on .NET Core 2.0 (for example), you can also run on the .NET Framework 4.6.1. Mono 5.4, Unity 2018.1 and more, because all of these runtimes implement the .NET Standard 2.0.

Of course, some features are always going to be very platform specific, so the standard started out small but has grown over time.

Moving to convergence and .NET.

Given that the later versions of the .NET Framework and .NET Core actually follow the same standard, the platforms are actually starting to become more and more similar.
They are becoming so similar that it no longer makes sense to maintain them separately. The next major version of both platforms is .NET 5. This is a new runtime which is the next version of .NET Core and the .NET Framework.

This means that the .NET Framework and .NET Core are going to converge into a single platform, which will be wonderful for developers and simplify a complex landscape.

But what does this mean if you have .NET Framework projects? How do we modernise, and do we have to make trade-offs around compatibility?

The Challenge: Modernisation and Compatibility

I have a number of projects which target the .NET Framework. On all of these projects I have had multiple requests to migrate to the .NET Core, but I have had to hold off on this work until I could really understand in detail a few things:

What would this mean for consumers of the libraries? Would they have to change the platform they use? Could this break things for them?
What would this mean for developers on the platform? Would they need to change their development environment? Would this cause problems?
What would this mean for maintainers of the libraries? Would this greatly increase build and deployment complexity?

Finally I have found the time to be able to start to address these issues in detail - hopefully the learnings will be useful to anyone who is maintaining a .NET codebase and thinking about the future.

The Modernisation Process - Introducing our two Villains

There are two key projects I wanted to modernise. They are both reasonably well used, complex, and have some potentially serious complexities for multi-platform builds.

SharpGL is a library that allows developers to use OpenGL in .NET applications. The big challenge? OpenGL is cross platform, but SharpGL specifically provides an interface to the Windows version of OpenGL. Can this possibly be made more future-proof? Could it ever target other platforms?

SharpShell is a library that allows developers to build ‘Shell Extensions’ for Windows. Shell extensions are customisations to the Windows user interface, so would not be portable across platforms, but I still want to ensure that the project is future proof.

What would be the experience with these two projects? I have other .NET projects, but they are far less popular and much more simple, my instinct is that if I can work through the process with these projects, the others should be more straightforward.

These are the steps I’ve followed to modernise. I’ll finish the article with a summary of the key learnings.

Step 1 - Understand the Domain

I cannot stress this enough. In all meaningful technology work, understand the domain you are dealing with. A quick Google on how to migrate, or following the formal migration guide was not enough for me. I knew I had to actually understand, at a reasonably detailed level, the differences in the runtime, the trade-offs, the process, the complexity.

This article is the result of that work - sometimes writing about a topic is the best way to force yourself to learn it.

Making changes rapidly and waiting to see what the consequences are can often work for small projects, internal tools and so on, but for a library which is relied upon by others is not good for the community. The last thing I wanted to do was make changes which had unintended consequences for users. So making sure that I learnt about this space, how things work under the hood, and what the expected changes in the future are was critical.

Hopefully for others the process of understanding the domain will be a little easier with this article to cover the high level topics. During my actual process of writing and migrating, I went a lot deeper than this article goes.

The key document to follow to actually execute the migration is the excellent official .NET Framework to .NET Core Porting Guide.

Step 2 - Understand the Goal - Multi-Platform Builds

Given the understanding of the domain, it made it much easier to understand what the required steps would be. Essentially, all that would be needed would be to target a version of the .NET Framework which adheres to a recent version of the .NET Standard. Once this was done, in theory the project could be built for the .NET Framework and for .NET Core, and also be ready for the upcoming .NET 5 release.

Multi-platform builds are supported in Visual Studio 2019. These builds allow us to have a single codebase, but build libraries for multiple platforms (i.e. the .NET Framework and .NET Core). The resulting binaries can be packed as a single package, and when consumers install the package, the appropriate library is installed.

This introduces the first of the significant consequences - modernising your project means you must migrate it to Visual Studio 2019.

In the past, this might have been more of an issue, licenses for Visual Studio were expensive, and many organisations were locked onto specific versions for compatibility issues (or because they were slow to upgrade). This seems to be the case less often nowadays, but is still an important consideration.

My projects were using Visual Studio 2017. This is how the project properties looked:

Unsurprisingly the .NET Standard isn’t mentioned. Time to upgrade to 2019. While I installed it I could reminisce about the excitement of buying Visual C++ .NET Learning Edition:

And try and remember what is was like to be a 15 years old. I wonder if that box set is still kicking around somewhere, I want to see it again. So much has changed. But long install processes for Visual Studio haven’t, at least they kept that:

When installing, remember to enable the .NET Core features.

Step 3 - Migrate Projects “Leaf-wise”

As per the Porting Guide, we need to migrate each of the projects which make up the solution, starting with the ’leaves’ (projects which don’t depend on other projects) and then moving up the tree to the ‘root’ projects (projects which are depended on by others).

Visually, for a solution like SharpGL, that would mean the projects will need to be converted in the following order:

I was expecting each project to have quite different experiences:

SharpGL.Serialization is just a set of classes which load data from files. In theory, this library should become completely portable.
SharpGL.WPF and SharpGL.WinForms are specifically for Windows front-end technologies. I expected these to be able to be ported, but don’t expect them to work on other platforms (in the future there might be SharpGL.OSx, or SharpGL.Gnome, who knows)
SharpGL.SceneGraph is a set of classes which represent 3D scenes - things like lights, cameras, materials and so on. I expect some of this to ‘just work’, but things like image loading to perhaps need some tweaking.
SharpGL is just a wrapper around the Windows opengl32.dll library. I can’t imagine this working anywhere but Windows, but how would the project structure porting go and would it build?

The details on how to migrate a project are in the Porting Guide, but the general approach will be:

Attempt to convert to the latest project format with the try-convert tool
Re-target the project to the .NET Framework 4.7.2 (the first version which supports the .NET standard)
Repeat for projects which this project depends on, walking the tree of projects to the root
Run the Portability Analysis tool to see if there are APIs which are not available on certain platforms

This is how you project might look after migration, having run the try-convert:

Now we just need to edit the project files and change the line:

<TargetFramework>4.7.2</TargetFramework>

To:

<TargetFrameworks>netcoreapp2.0;netcoreapp3.0;netcoreapp3.1;net40;net45;net472</TargetFrameworks>

The targets you will choose will depend on the APIs you want to use. There is an Portability Analysis extension available which can build a portability report, here’s what one looks like:

This will also show the specific APIs which are not compatible with specific targets:

Now it’s time to move to the next step.

Step 4 - Refactor, Rinse, Repeat

This is the tricky part. You’ll now need to work out whether you want to remove API calls which are not portable, try and use alternatives, or conditionally compile the code for different platforms.

If you are using non-portable APIs you may need to use conditional blocks to execute different code depending on the framework used. The Target frameworks in SDK-style projects guide shows how to do this.

You may also have to manually edit the project file to ensure that certain dependencies are only used for certain targets. You solution file and dependencies may end up looking something like this:

Once you have reloaded the project you’ll see your dependencies can now be specified on a per-framework basis, and a build generates assemblies for each of the targets:

This process might be simple, or complex, depending on the nuances of your project. For me it was fairly iterative - starting by targeting only net40 (the original target framework which I’d used), then adding more and more targets.

Some targets will simply not be possible - for example .NET Core only supports WinForms and WPF from .NET Core 3.0 onwards; you won’t be able to build a WinForms or WPF assembly which targets a lower version, the framework doesn’t support it.

Step 5 - Update Your Builds

At this stage, having fixed compatibility issues, you should have code which builds in Visual Studio.

Now I would recommend porting all of your build code to use the dotnet build system. This is going to maximise the portability and future-proof your project, you’ll be able to run the builds on multiple platforms and are using the preferred standard tool (msbuild will essentially become legacy).

The way I like to structure things personally is have a set of scripts which you can run to build, test and package the code locally. You can then call these scripts from you CI tool of choice to automate things, but still keep the logic in your own code, rather than hidden away in a build system.

For example, in my SharpGL project I have the following scripts:

Script	Usage
`config.ps1`	Ensure your machine can run builds by installing necessary components such as `nunit`. Should only need to be run once.
`build.ps1`	Build all solutions.
`test.ps1`	Run all tests, including those in samples.
`coverage.ps1`	Create a coverage report. Reports are written to `./artifacts/coverage`
`pack.ps1`	Create all of the SharpGL NuGet packages, which are copied to `./artifacts/packages`.

I updated my scripts to use the dotnet tool. For example, the ‘build’ script looks something like this:

# Run the build, hiding the documentation warnings for pinvoke code.
$buildCommand ="dotnet msbuild $PSScriptRoot\SharpGL.sln -noWarn:CS1591 -noWarn:CS1573 -t:Rebuild -p:Configuration=Release"
Write-Host "Running: ""$buildCommand"""
Invoke-Expression $buildCommand

And the ‘pack’ script looks like this:

dotnet pack --no-restore --no-build "$PSScriptRoot/Core/SharpGL/SharpGL.csproj" -c:Release
dotnet pack --no-restore --no-build "$PSScriptRoot/Core/SharpGL.SceneGraph/SharpGL.SceneGraph.csproj" -c:Release
dotnet pack --no-restore --no-build "$PSScriptRoot/Core/SharpGL.Serialization/SharpGL.Serialization.csproj" -c:Release
dotnet pack --no-restore --no-build "$PSScriptRoot/Core/SharpGL.WinForms/SharpGL.WinForms.csproj" -c:Release
dotnet pack --no-restore --no-build "$PSScriptRoot/Core/SharpGL.WPF/SharpGL.WPF.csproj" -c:Release

The actual scripts are a little more complex. But the key thing here is that I can run any part of the CI/CD process locally (to test, debug and so on) or on a CI/CD platform.

You will most likely have to conditionally reference certain components. The dependency for net40 might be different to that for netcoreapp3.0. You’ll see that in many of my project files there is now code like this:

<ItemGroup>
 <Reference Include="System.Design" Condition="'$(TargetFramework)' == 'net40'" />
 <Reference Include="System.Design" Condition="'$(TargetFramework)' == 'net45'" />
 <Reference Include="System.Design" Condition="'$(TargetFramework)' == 'net472'" />
 <Reference Include="System.Windows.Forms" Condition="'$(TargetFramework)' == 'net40'" />
 <Reference Include="System.Windows.Forms" Condition="'$(TargetFramework)' == 'net45'" />
 <Reference Include="System.Windows.Forms" Condition="'$(TargetFramework)' == 'net472'" />
</ItemGroup>
<ItemGroup>
 <PackageReference Include="Microsoft.CSharp" Version="4.7.0" Condition="'$(TargetFramework)' == 'netcoreapp3.0'" />
 <PackageReference Include="Microsoft.CSharp" Version="4.7.0" Condition="'$(TargetFramework)' == 'netcoreapp3.1'" />
 <PackageReference Include="Microsoft.CSharp" Version="4.7.0" Condition="'$(TargetFramework)' == 'net45'" />
 <PackageReference Include="Microsoft.CSharp" Version="4.7.0" Condition="'$(TargetFramework)' == 'net472'" />
 <PackageReference Include="System.Data.DataSetExtensions" Version="4.5.0" Condition="'$(TargetFramework)' == 'netcoreapp3.0'" />
 <PackageReference Include="System.Data.DataSetExtensions" Version="4.5.0" Condition="'$(TargetFramework)' == 'netcoreapp3.1'" />
 <PackageReference Include="System.Data.DataSetExtensions" Version="4.5.0" Condition="'$(TargetFramework)' == 'net45'" />
 <PackageReference Include="System.Data.DataSetExtensions" Version="4.5.0" Condition="'$(TargetFramework)' == 'net472'" />
</ItemGroup>

In my case quite a bit of trial and error was needed to find the appropriate references for each platform.

Step 6 - Simplify!

One benefit I have found during this process is that you can simplify your projects. You no longer need any kind of ‘automated NuGet restore’ functionality. This means you can remove code like this from your project files:

<Import Project="$(SolutionDir)\.nuget\NuGet.targets" Condition="Exists('$(SolutionDir)\.nuget\NuGet.targets')" />
<Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
 <PropertyGroup>
 <ErrorText>This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText>
 </PropertyGroup>
 <Error Condition="!Exists('..\packages\NUnit.3.11.0\build\NUnit.props')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\NUnit.3.11.0\build\NUnit.props'))" />
 <Error Condition="!Exists('..\packages\NUnit3TestAdapter.3.10.0\build\net35\NUnit3TestAdapter.props')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\NUnit3TestAdapter.3.10.0\build\net35\NUnit3TestAdapter.props'))" />
</Target>

You can also remove your project.json as all of the data is now in the csproj file.

Another nice update is that you no longer need to maintain an AssemblyInfo.cs file; you can keep all of your assembly metadata in the csproj file.

Finally, you can almost certainly remove any nuspec files - all NuGet packaging data can also be embedded in the csproj file. For example, here’s what my SharpShell project metadata looks like:

<Project Sdk="Microsoft.NET.Sdk.WindowsDesktop">
 <PropertyGroup>
 <TargetFrameworks>netcoreapp2.0;netcoreapp3.0;netcoreapp3.1;net40;net45;net472</TargetFrameworks>
 <OutputType>Library</OutputType>
 <!-- The following properies are used to manage how the project is packaged. -->
 <PackageId>SharpShell</PackageId>
 <Copyright>Copyright (c) Dave Kerr 2020</Copyright>
 <PackageProjectUrl>https://github.com/dwmkerr/sharpshell</PackageProjectUrl>
 <RepositoryUrl>https://github.com/dwmkerr/sharpshell</RepositoryUrl>
 <Version>3.1.1.0</Version>
 <Authors>Dave Kerr</Authors>
 <Company>Dave Kerr</Company>
 <PackageTags>Shell;SharpShell;COM;Context Menu;Icon Handler</PackageTags>
 <Description>SharpShell is a framework that lets you build Windows Shell Extensions using .NET Core or the .NET Framework.</Description>
 </PropertyGroup>
 <!-- ...snip... -->
</Project>

This helps to keep a lot of the project dependency and property data in one place and is probably more convenient for many users.

You can see the Pull Request for SharpGL to see how the project files were updated in this case. You can also see the SharpShell Pull Request. The SharpShell version is still work in progress at the time of writing.

Step 7 - Test, Test, Test

Now for the fun part. You are going to really have to test the new packages on each platform. Sadly, this kind of migration is not something which will have issues exposed via unit tests, you’ll need to create test projects which import your packages, ideally for each platform, and make sure they work. There could be runtime errors, particularly if you have made mistakes with the references.

Many issues will be caught at compile time - some will not.

Here’s a screenshot of me having fun trying out the .NET Framework 4 package for WinForms, and the .NET Core 3.1 package for WPF:

How you test your packages will be very dependent on what you are building. If it is highly platform specific then you will likely have to do lots of testing. If it is fairly self-contained code then you might be able to get away with some basic smoke testing.

Step 8 - Document Compatibility

If you are supporting multiple platforms and frameworks, it’s going to be a lot of help to consumers of your code if you can be very clear about what is supported.

This may be more complex than you think. Your library may run fine as part of a .NET Core Console Application on Windows - but does it work on MacOS? What about Linux?

Here’s a screenshot I would never have imagined when I started the SharpGL project - a terminal application running on MacOS which is using the SharpGL.Serialization library to load geometry from a file:

Now of course for something like SharpGL to run on a Mac or Linux, a lot more work would be needed. SharpGL is at its core nothing more than a wrapper around opengl32.dll on Windows, on other platforms there are no DLLs, but OpenGL is still available. So support is possible, but not ready yet. So at this stage, docmenting what you know works as well as what doesn’t will be really helpful.

You might also want to preserve your ‘pre-migration’ code in a separate branch, in case you have users who for some reason have issues migrating and need to use an older version. For SharpGL, I updated the project page to indicate compatibility, what has been tested and so on:

The Key Learnings

Here are the key learnings which stood out for me as I worked on migration of these projects.

Consumer Experience

If you are careful, you don’t have to break anything for consumers - with multi-targeting you can still target older frameworks.
You can potentially greatly increase the compatibility of your projects by offering support for .NET Core.

Developer Experience

You need to upgrade to Visual Studio 2019…
…however, you can use Visual Studio for Mac or even the command-line to build across many platforms.

Maintainer Experience

You will have a much larger set of potential consumers, but you will likely find bugs which are framework or platform specific.
You will likely need to work on migrating your project files and use the latest dotnet tooling.
You should be careful to document known compatibility issues.

All in all, the process was less painful than I expected. Now that this work is done I can focus on more exciting things, such as potentially getting projects like SharpGL working on Linux or MacOS, which is much more exciting.

As always, questions, comments, suggestions, rants, anything are welcome!

The pull request which migrates the SharpGL project and SharpShell projects are below:

Useful References

Observations, tips and tricks for the CKA certification

Thu, 04 Jun 2020 00:00:00 +0000

In this article I’ll share some observations, tips and tricks for the Linux Foundation’s “Certified Kubernetes Administrator certification and exam.

I’ve been operating Kubernetes in multiple environments for a few years now. I thought this would be an easy certification to get, but I was surprised by how hard it was!

I took this exam without doing any formal training, I mostly focused on the areas of the curriculum which I knew I was a little weak at. The task-based structure for the exam I thought was really excellent. It took me two attempts to pass, and I learnt a few things along the way.

Here I’ll share some thoughts on the certification which hopefully will be useful if you are considering taking it!

Tip: Do the right Certification!
Tip: Understand the Format!
Tip: Know your Vim
You need to know the architecture of Kubernetes
Tip: You Need to know Linux Sysadmin
Tip: “Dry Run” is your friend
Tip: Know how to troubleshoot networking
Tip: Nail the easy questions quickly
That’s it!

Tip: Do the right Certification!

The CKA exam tests administration and operation skills and techniques for Kubernetes. If you have set up and administered clusters before, this will likely not be too challenging. But if you’ve never set up a cluster by hand, troubleshot weird issues, fixed clusters and so on, then this is likely going to be very hard.

There is a certification which is much more geared towards developers who use Kubernetes, but don’t necessarily administer it - that’s the CKAD exam and might be the one to take if you are not too familiar with system administration.

Tip: Understand the Format!

This is not a multiple choice question exam. It’s a task based exam, meaning you have about 22 or so specific tasks to complete, in a web browser which has a terminal connected to a cluster.

It is open-book - meaning that you can use the Kubernetes Documentation during the exam. It’s not a memory test of specific flags for commands or whatever, it will really require you to work with a running cluster. This means you’ll have to be pretty familiar with kubectl, kubeadm and also Linux in general!

Tip: Know your Vim

In the two exams I took, nano was available. But if you are using nano to work with files you may struggle for time.

I spent a lot of time in vim in the exam. vim is my main text editor for day to day work, so I’m fairly familiar with it. Knowing how to quickly copy a file (lets say for example a file which represents a deployment) and quickly manipulate the text in it will be crucial. Make sure you are going to be using a text editor which you can be efficient in!

You won’t be using a graphical text editor to work with files, so being competent in a terminal editor like vim or emacs could make a big difference. Of course you could install your favourite text editor, but you won’t be able to use a graphical editor like VS Code.

Also, as in most Linux distributions, screen is available out of the box, and tmux can also be installed. If you are familiar with either of these terminal mutliplexers it could save you a tonne of time, for example being able to run watch -n 5 -d kubectl get pods in one pane while applying resources in another.

You need to know the architecture of Kubernetes

This exam will require you to deal with trivial tasks such as running a deployment or creating a volume. But the questions which focus on that tend to only count for one or two percent of the overall grade each. Questions which deal with troubleshooting actual Kubernetes issues could count for six or seven percent each.

This means you need to know how Kubernetes is architecture. The kubelet which runs on nodes, the API server, the etcd store, all of these things you have to understand how they work and how they fit together.

The online documentation covers the architecture in detail, here’s the best place to start:

https://kubernetes.io/docs/concepts/overview/components/

You will need to know how the control plane works, how nodes communicate, how transport of messages works and is secured if you are going to have a chance at dealing with the harder questions.

Tip: You Need to know Linux Sysadmin

If you are not familiar with systemctl, journalctl, apt, systemd units and how the core Kubernetes components are configured, you’ll really struggle.

Look over the CNCF curriculum - expect to not just have to know how to deal with ‘happy path’ situations, but also broken clusters, incorrect configuration and so on.

Tip: “Dry Run” is your friend

One thing which helped me a lot in my second attempt at the exam was the --dry-run flag. Before you create resources or change anything, run the operation with the --dry-run flag and see whether the output is what you would expect.

This is a quick and easy way to see the changes to the cluster which you are going to apply - and troubleshoot them - before making any actual changes.

Tip: Know how to troubleshoot networking

Networking in Kubernetes is complex. You must be able to troubleshoot networking issues in the cluster to be able to deal with the more complex tasks.

This means that you should know how to be able to run typical networking tools like dig, nslookup, telnet etc, in the cluster itself.

If you are not familiar with these tools you might need to take an online course in Kubernetes or Linux Networking Administration before considering this certification. The Linux Certified Systems Administrator training would be a good place to start.

If you have taken the Docker Certified Associate exam then some of this should be familiar. If you are not very familiar with how Docker itself works, you’ll likely struggle with Kubernetes.

Tip: Nail the easy questions quickly

There are a lot of tasks which only count for one or two percent each; these ones you should be able to complete in a few minutes. You’ll need all the time in the exam to work on the really hard questions which deal with diagnosing and fixing cluster issues.

Know your core Kubernetes concepts; if you have done the CKAD exam you should be good, if not, check the curriculum and make sure you can quickly complete all of the trivial tasks without wasting too much time.

That’s it!

Hopefully this was helpful! Good luck if you are taking the exam and hopefully you’ll find it a challenging but rewarding experience. I’ve taken many exams over the years but this was one of the most challenging, but also one of the most enjoyable, I really felt like it was testing practical techniques rather than your ability to just remember random commands and flags.

As always, if you have any comments or questions, please just add them in the section below!

Supercharge your Java Projects with Conventional Commits, Semantic Versioning and Semantic Releases

Sun, 17 May 2020 00:00:00 +0000

In this article we’ll look at a few simple techniques which can really supercharge your Java project and make them much easier to work with!

Semantic Versioning
Conventional Commits
Time for Magic
- Enforcing Conventional Commits with Git Hooks
- How the Hook Works
Creating the Initial Release
Go Forth And DevOps
The Gradle Version
That’s It

tl;dr If you know the concepts, then just jump straight to my fork of standard-version at github.com/dwmkerr/standard-version. It adds support for Java projects. I am currently trying to get it into the mainline, so if you like this, please comment on the Pull Request here. tl;dr end!

Semantic Versioning

First, let’s talk about the idea of a Semantic Version. A semantic version is nothing more than a versioning scheme you will probably be familiar with, where versions look like this:

1.2.3

The only thing special about a Semantic Version is that we give a very specific meaning to each part of the version. In short:

1 is the major part of the version
2 is the minor part of the version
3 is the patch part of the version

Now we give semantics (meaning and context) to these parts:

Major

A major version number change means something big has changed, and the API of the software is different to the earlier version. Essentially, this is a potentially breaking change, so you should only use this new version after carefully reading about the changes.

Minor

A minor version number change means that something has been added or changed, which affects the functionality of the code, but in a non breaking way. An example would be the addition of a new API. That won’t affect existing users, so they can generally safely upgrade minor versions without too much risk.

Patch

A patch version number change means something really inconsequential to the user of the code has changed. It might be new documentation, better logging, but it is generally not a functional change.

Why Does This Matter?

If we have Semantic Versions, we can be a lot more sure about when it is safe to upgrade. If we see a major version change, we know we need to be careful. Minor changes might need attention, and patches are almost always going to be safe.

Managing dependencies and keeping them up to date is hard in software development, and one of the reasons people are wary of updating dependencies is that they don’t know if they upgrade will break their code.

Semantic Versioning tries to bring a little order to this chaotic world.

The Semantic Versioning Specification

There is a detailed specification for semantic versioning, which also covers more sophisticated cases, you can find it here:

https://semver.org/

I’d suggest this is recommend reading for any software engineer!

Using Semantic Versions

Now the easiest way to start with semantic versioning is to simply adhere to the spec! For example, if you make a change which could break something for users, bump the major part of the version.

But, things aren’t all that easy…

The Challenge of Semantic Versions

The challenge is this. Imagine you are cutting a new release of your code and many people have contributed. Some bug fixes, some patches, some documentation. How do you look through all of those changes and decide how to appropriately change the version number?

To solve this problem, say hello to Conventional Commits.

Conventional Commits

If you have a commit history like this:

Updated the users API
Bugfix
trying the build again, got it working
Bugfix: [JIRA-21] fixed that issue
you can now get user's friends with this change

Then it is very hard to reason about what is going on. What about if the commit history looked like this?

feat(users): [#12] fetching users returns their avatar url
fix(users): [#45] display names with emojis return correctly
build(cicd): update the expired deploy key
fix(docs): [#22] fix broken links to the javadocs
feat(users): [#49] users api optionally returns friends, non-breaking

It’s much easier to see what each change means, at least at a high level.

By having some kind of standard for commit messages, we can do a lot. We can:

Classify changes by type (such as a feature or fix)
Include a clear description of the change
Use a convention to indicate a breaking change
Link to a ticketing system

Just like semantic versioning, conventional commits have a specification too:

https://www.conventionalcommits.org/en/v1.0.0/

Time for Magic

Now if we have conventional commits, and want to use semantic versions, we can actually skip the whole process of looking over a commit history to create a new semantic version - we can automate it.

We can even automate the process of creating a ‘changelog’, a list of each change which comes in each version. There’s an excellent library which does this, called standard-version:

https://github.com/conventional-changelog/standard-version

It’s maintained by the same group behind conventional commits. The only problem? It only works for JavaScript projects (unless you are willing to write custom code which can be complex).

But I’ve updated the library to support Maven projects and Gradle projects, so you can use it for Java now as well!

Let’s see it in action. Here’s a very simple Java library built with Maven:

https://github.com/dwmkerr/java-maven-standard-version-sample

This library has no changelog, no tags, no version data at all except for 0.1.0 in the pom.xml file.

Now if I was to clone the library, make a change and make a commit, which didn’t follow the conventional commit spec, we’ll just see the usual success message:

This is a problem; we want to enforce conventional commits.

Enforcing Conventional Commits with Git Hooks

Git has a powerful ‘hooks’ facility, which let you run logic at key points in operations. This is a massive topic on its own, so we’re not going to go into lots of details, but if you are interested you can read about them here:

https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks

Now the issue with Git Hooks is that they are per user - if I add a hook to my .git folder, no one else will get it. We want the same hooks for all users.

There are a few ways to get around this. You can set up server side hooks (which could reject a push if it has an invalid commit message), but this isn’t easy to do (and with some providers, like GitHub for public projects, not even available as an option). Also, we want fast feedback, so if I make a bad commit message, it fails straight away and I can fix it.

The way I suggest getting around this is this:

Create a .githooks folder in your repo
Instruct people to configure the git repo to look for hooks there

That way there are no global changes, only project specific ones. We still need to make sure the developer sets up the hooks though! You’ll notice in my sample project’s README.md file the first thing I do is instruct people to setup the hooks:

git config core.hooksPath .githooks

Let’s see how the same operation would look if we’d setup the hook first:

Our hook has fired off and told us we’ve not used a conventional commit message - it even let’s us know where to go to find out more.

Let’s try a message which should work, as it meets the standard:

Awesome! We’ve been informed that our message meets the standard (useful to actually remind us that this is being checked!) and the commit has succeeded!

Remember; we only need to setup the hooks once - it’s a one time activity.

How the Hook Works

Hooks are just shell scripts. You can write them in Ruby, Python, whatever. I have written this one in pure Bash because it’s really just checking a regex, which Bash is more than capable of. Also, I can’t be sure the developer will have Ruby or another tool on their machine.

The hook is as simple as this:

#!/usr/bin/env bash

# Create a regex for a conventional commit.
convetional_commit_regex="^(build|chore|ci|docs|feat|fix|perf|refactor|revert|style|test)(\([a-z \-]+\))?!?: .+$"

# Get the commit message (the parameter we're given is just the path to the
# temporary file which holds the message).
commit_message=$(cat "$1")

# Check the message, if we match, all good baby.
if [[ "$commit_message" =~ $convetional_commit_regex ]]; then
 echo -e "\e[32mCommit message meets Conventional Commit standards...\e[0m"
 exit 0
fi

# Uh-oh, this is not a conventional commit, show an example and link to the spec.
echo -e "\e[31mThe commit message does not meet the Conventional Commit standard\e[0m"
echo "An example of a valid message is: "
echo " feat(login): add the 'remember me' button"
echo "More details at: https://www.conventionalcommits.org/en/v1.0.0/#summary"
exit 1

The only really tricky bit is the regex, and the weird \e[32 type characters which are used to set the colours. You might find it easier to write your hooks in a proper programming language - and for anything more complex I’d suggest that makes far more sense! But if a bit of Bash will do the trick, there’s nothing wrong with that.

As a side-note, if you are into Bash and the shell, check out my online Effective Shell book. git config core.hooksPath .githooks

Creating the Initial Release

Now the chances are, if you are interested in this technique, you’ve probably got an existing project you want to use it on. It probably doesn’t have a changelog or conventional commits. That’s OK, just start from now.

Here’s how we’d start using the standard-version library to manage our versions. I’ve added a new API to the release branch (to keep master clean for people reading the sample) and committed it.

Now lets actually create our changelog:

npx @dwmkerr/standard-version --first-release --packageFiles pom.xml --bumpFiles pom.xml

Now it’s a pain I know, but you need Node.js installed for this to work. The standard-version library is built on node, that’s what is used to do all of the logic around writing a changelog and working out what the version bump should be. You also have to use my fork @dwmkerr/standard-version rather than the main version, because at the time of writing my pull request which adds support for pom.xml files is not yet merged.

What has happened here is that the standard-version tool has not changed the version number. We told it this is the first-release, meaning we haven’t published yet, so there’s no need to create a new number. What is has done is given us a changelog and told use how to push the tags and code. If we push, we can now see the changelog:

See how we get a changelog showing the changes, the version and the date? We even have links to the commits for each key change!

If we’d linked the message to GitHub Issue numbers it’d automatically have links to the issues too!

Now in this code I deliberately made a mistake - the test for the Goodbye api is a copy and paste of the Hello test! And the Goodbye api has a spelling mistake. Let’s fix this and cut a new release.

I’ve made the change on the release branch, now I’ll run standard-version again:

npx @dwmkerr/standard-version --packageFiles pom.xml --bumpFiles pom.xml

Note that there was no need for the --first-release flag.

Now this time, a new version has been generated. This was a fix commit, so it has made it a minor version bump. If we needed to make it a breaking change, we can use a message with an exclamation after the type, such as fix(goodbye)!: fix the typo. Check the standard-version docs for more about this.

Finally, let’s look at our new changelog:

We have even more info now - we have a link to the tag. This is incredibly useful for managing releases.

The icing on the cake? Let’s look at the pom.xml:

Note that the version has been updated. standard-release is keeping our Git Tags and our Java Library Version numbers automatically in sync.

Once you’ve started doing this and seen it in action for a while, you’ll wonder how you lived without it!

Go Forth And DevOps

This is just the beginning! Think of all the cool things we can do with this in place, here’s just a few:

Update our build pipeline so that when we merge into master we automatically run standard-version
Update our build pipeline so that when a new version tag is added, we automatically publish the library
Send out a slack notification with the changelog when a new version is committed
Share the changelog with our consumers as our libraries are updated

With these basic building blocks:

Conventional Commits
Semantic Versioning
Enforcing of Commit Standards
Usage of the standard-release tool

We have created a very powerful way to manage what is actually a highly complex process. We’ve introduced almost no additional complexity, just a few guidelines for developers.

The Gradle Version

It’s basically the same technique for Gradle, you just tell standard-version to hit your build.gradle file;

npx @dwmkerr/standard-version --packageFiles build.gradle --bumpFiles build.gradle

There’s an accompanying sample project at:

github.com/dwmkerr/java-gradle-standard-version-sample

This is the same as the Maven version in that the master branch has no standard-version code or changelogs, just open the release branch to see what it looks like after we’ve applied the same techniques as we did to the Maven version.

That’s It

There’s a whole world of libraries for this. commitizen which helps you write conventional commit messages for example. But I found very little for Java. If you find this useful, please do chip in on the pull request here:

https://github.com/conventional-changelog/standard-version/pull/591

As it would be great to add it to the mainline. I’m also just finishing off the update which adds support for Gradle.

As always, questions, comments, suggestions, rants, anything are welcome!

Effective Shell for Beginners

Tue, 21 Jan 2020 00:00:00 +0000

I have rebuilt my “Effective Shell” series as an online book - it’s available now on:

https://effective-shell.com

The whole site is built from a GitHub repo at github.com/dwmkerr/effective-shell. It is open for contributions, changes, issues and suggestions. I’ve also added a comment section to each page to get input.

To keep the material as accessible as possible, I have added a new section for beginners, to help anyone who has not used a shell before. It goes over who the book is useful for, what the shell is, and how to set up your computer to work through the material:

All comments and suggestions are welcome!

Effective Shell Part 7: The Subtleties of Shell Commands

Tue, 25 Jun 2019 07:25:23 +0000

In this chapter, we’ll take a look at the various different types of shell commands that exist and how this can affect your work.

By the end of this chapter, you might even be able to make sense of the horrifying and perfectly syntactically valid code below:

which $(where $(what $(whence $(whereis who))))

What Are Commands?

This is really important to understand! A command in a shell is something you execute. It might take parameters. Generally it’ll have a form like this:

command param1 param2

We’ve already seen many commands during this series:

ls # Show the contents of the current directory
cd ~ # Move to the user's home
cat file.txt # Output the contents of 'file.txt' to stdout

But to be an effective shell user, you must understand that not all commands are created equal. The differences between the types of commands will affect how you use them.

There are four types of commands in most shells:

Executables
“Built-Ins” (which we’ll just call builtins from now on)
Functions
Aliases

Let’s quickly dig in and see a bit more.

Executables - Programs

Executables are just files with the ’executable’ bit set¹. If I execute the cat command, the shell will search for an executable named cat in my $PATH. If it finds it, it will run the program.

$ cat file.txt
This is a simple text file

What is $PATH? $PATH is the standard environment variable used to define where the shell should search for programs. If we temporarily empty this variable, the shell won’t find the command:

$ PATH="" cat file.txt
bash: cat: No such file or directory

Normally your $PATH variable will include the standard locations for Linux programs - folders such as /bin, /sbin, /usr/bin and so on².

If you were to print the variable, you’d see a bunch of paths (they are separated by colons; I’ve put them on separate lines for readability):

/usr/local/bin
/usr/bin
/bin
/usr/sbin
/sbin

The shell will start with the earlier locations and move to the later ones. This allows local flavours of tools to be installed for users, which will take precedence over general versions of tools.

There will likely be other locations too - you might see Java folders, package manager folders and so on.

Executables - Scripts

Imagine we create a text file called dog in the local folder:

#!/bin/sh
echo "🐶 woof 🐶"

If we make the file executable, by running chmod +x dog³, then we can run this just like any other program - as long as we tell the shell to look for programs in the current directory:

$ PATH="." dog
🐶 woof 🐶

More common would be to run the program by giving a path:

$ ./dog
🐶 woof 🐶

Or just move it to a standard location that the shell already checks for programs:

$ mv dog /usr/local/bin
$ dog
🐶 woof 🐶

The point is that executables don’t have to be compiled program code. If a file starts with #! (the ‘shebang’), then the system will try to run the contents of the file with the program specified in the shebang.

We will look at shebangs in greater detail in a later chapter.

Builtins

OK, so we’ve seen executables. What about a command like this?

local V="hello" echo $V

You will not find the local executable anywhere on your system. It is a builtin - a special command built directly into the shell program.

Builtins are often highly specific to your shell. They might be used for programming (local for example is used to declare a locally scoped variable), or they might be for very shell-specific features.

This is where we need to take note. As soon as you are running a builtin, you are potentially using a feature that is specific to your shell, rather than a program that is shared across the system and can be run by any shell.

Trying to programmatically execute local as a process will fail - there is no executable with that name; it is purely a shell construct.

So how do we know if a command is a builtin? The preferred method is to use the type command:

$ type local
local is a shell builtin

The type command (which is itself a builtin!) can tell you the exact type of shell command.

Interestingly, you might be using more builtins than you think. echo is a program, but most of the time you are not executing it when you are in a shell:

$ type -a echo
echo is a shell builtin
echo is /bin/echo

By using the -a flag on type to show all commands that match the name, we see that echo is actually both a builtin and a program.

Many simple programs have builtin versions. The shell can execute them much faster.

Some commands are a builtin so that they can function in a sensible manner. The cd command changes the current directory - if we executed it as a process, it would change only the directory for the cd process itself, not the shell, making it much less useful.

Builtins will vary from shell to shell, but many shells are ‘Bash-like’ - meaning they will have a set very similar to the Bash builtins, which you can see here:

https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html

As should be familiar from Part 3: Getting Help, you can get help for builtins:

$ man source # source is a builtin
BUILTIN(1) BSD General Commands Manual BUILTIN(1)

NAME
 builtin, !, %, # ...snip...

SYNOPSIS
 builtin [-options] [args ...]

However, the manual will not show information on specific builtins, which is a pain. Your shell might have an option to show more details - for example, in Bash you can use help:

$ help source
source: source filename [arguments]
 Read and execute commands from FILENAME and return. The pathnames
 in $PATH are used to find the directory containing FILENAME. If any
 ARGUMENTS are supplied, they become the positional parameters when
 FILENAME is executed.

But remember: help is a builtin; you might not find it in all shells (you won’t find it in zsh, for example). This highlights again the challenges of builtins.

Functions

You can define your own shell functions. We will see a lot more of this later, but let’s show a quick example for now:

$ restart-shell () { exec -l "$SHELL" }

This snippet creates a function that restarts the shell (quite useful if you are messing with shell configuration files or think you might have irreversibly goofed up your current session).

We can execute this function just like any command:

$ restart-shell

And running type will show us that this is a function:

$ type restart-shell
restart-shell is a function
restart-shell ()
{
 exec -l $SHELL
}

Functions are one of the most powerful shell constructs we will see; they are extremely useful for building sophisticated logic. We’re going to see them in a lot more detail later, but for now it is enough to know that they exist, and can run logic, and are run as commands.

Aliases

An alias is just a shortcut. Type in a certain set of characters, and the shell will replace them with the value defined in the alias.

Some common commands are actually already aliases - for example, in my zsh shell, the ls command is an alias:

% type -a ls
ls is an alias for ls -G
ls is /bin/ls

I make sure that when I use the ls command, the shell always expands it to ls -G, which colours the output.

We can quickly define aliases to save on keystrokes. For example:

$ alias k='kubectl'

From this point on, I can use the k alias as shorthand for the kubectl command.

Aliases are far less sophisticated than functions. Think of them as keystroke savers and nothing more, and you won’t go far wrong. Aliases are not portable across shells and have certain behaviours which can make them problematic to work with, there will be an entire chapter dedicated to alisases coming up in the series.

So What?

So we now hopefully have a greater understanding of the variety of shell commands. Not all commands are executables, not all of the commands we think are executables necessarily are, and some commands might be more sophisticated.

As a shell user, the key things to remember are:

Executables are ‘safe’ - they are programs your system can use; your shell just calls out to them.
Builtins are very shell-specific and usually control the shell itself
Functions are powerful ways to write logic but will normally be shell-specific.
Aliases are conveniences for human operators, but only in the context of an interactive shell.

To find out how a command is implemented, just use the type -a command:

$ type -a cat
cat is /bin/cat

More than You Need to Know

OK, for the masochistic few, you might be wondering about all of the other commands and utilities you may have seen that can tell you about programs and commands:

what
whatis
which
whence
where
whereis
command
type

A lot of these are legacy and should be avoided, but for completeness sake, we’ll go through them.

`what`

what reads out special metadata embedded in a program, generally used to identify the version of source code it was built from:

$ what /bin/ls
/bin/ls
 Copyright (c) 1989, 1993, 1994
 PROGRAM:ls PROJECT:file_cmds-272.220.1

There should be almost no circumstance in which you need to use it in your day-to-day work, but you might come across it if you meant to type whatis.

`whatis`

whatis searches a local help database for text. This can be useful in tracking down manual pages:

$ whatis bash
bash(1) - GNU Bourne-Again SHell
bashbug(1) - report a bug in bash

But I can’t imagine it will be a regularly used tool by most users.

`which`

which will search your $PATH to see whether an executable can be found. With the -a flag, it will show all results.

$ which -a vi
/usr/local/bin/vi
/usr/bin/vi

which originated in csh. It remains on many systems for compatibility but in general should be avoided due to potentially odd behaviour⁴.

`whence`

whence was added to the Korn shell. You are unlikely to use it unless you are on systems using ksh. zsh also has this command, but it should be avoided and considered non-standard.

% whence brew
/usr/local/bin/brew

`where`

This is a shell builtin that can provide information on commands, similar to type:

% where ls
ls: aliased to ls -G
/bin/ls

However, type should be preferred, as it is more standard.

`whereis`

whereis is available on some systems and generally operates the same as which, searching paths for an executable:

% whereis ls
/bin/ls

Again, type should be preferred for compatability.

`command`

command is defined in the POSIX standard, so should be expected to be present on most modern systems. Without arguments, it simply executes a command. With the -v argument, you get a fairly machine-readable or processable response; with the -V argument, you get a more human readable response:

% command -v ls
alias ls='ls -G'
% command -V ls
ls is an alias for ls -G

command can be useful in scripts, as we will see in later chapters.

`type`

type is part of the Unix standard and will be present in most modern systems. As we’ve already seen, it will identify the type of command as well as the location for an executable:

% type -a ls
ls is an alias for ls -G
ls is /bin/ls

This command can also be used to only search for paths:

% type -p ls
ls is /bin/ls

Summary

In summary, avoid anything that starts with ‘w’! These are legacy commands, generally needed only when working on older Unix machines. type or command should be used instead.

Footnotes

We will cover permissions and modes in later chapters. ↩︎
Why these names and locations? It’s a long story. The best place to start if you are intersted is the Filesystem Hierarchy Standard. ↩︎
chmod changes the mode of a file; +x means ‘add the executable bit’. This tells the operating system the file can be executed. ↩︎
Stack Exchange: Why not use “which”? What to use then? ↩︎

Effective Shell Part 6: Everything You Don't Need To Know About Job Control

Mon, 10 Jun 2019 08:26:33 +0000

Job control is a feature of most shells, which is generally not particularly intuitive to work with. However, knowing the basics can help prevent you from getting yourself into a tangle, and can from time to time make certain tasks a little easier.

In this chapter, we’ll look at the main features of job control, why it can be a problematic, and some alternatives.

What Is Job Control?

Let’s start with an example. I am building a simple web page. It has one index.html file, one styles.css file, and one code.js file. The index.html file looks like this:

<html>
 <head>
 <title>My New Project</title>
 <link rel="stylesheet" type="text/css" href="styles.css">
 <script src="code.js"></script>
 </head>
 <body>
 <!-- Snip... -->
 </body>
</html>

Opening the file in a browser doesn’t quite work, as it won’t load the code or the styles. We need a web server to serve styles and code.

A super-useful one-liner to run a web server on any machine with Python installed is:

python -m SimpleHTTPServer 3000

In fact, this is so useful that I normally alias this command, so that I can just type serve. We’ll see aliases in a later chapter.

For now, if we run this command (you can get the three sample files here if you want to try this yourself), then we can open the webpage in a browser, with the styles and code loaded:

We can also see that the server has served the HTML, JavaScript, and CSS files:

All well and good so far.

The Problem

Let’s say we want to now continue using our shell, maybe to edit the website with a terminal editor like Vim or Emacs, or we want to zip up the site, or just run any shell command¹.

We have a problem. The python process is still running - it’s serving the website. Our shell is essentially useless, until we stop the server. See what happens when I try to edit a file:

In the example above, I try to run vi, but nothing is happening. Standard input is not being read by the server and not being interpreted by the shell.

I have to kill the server by hitting Ctrl+C (which sends a SIGINT² - we’ll see more about signals later), clear my screen to get rid of all of the error messages, then start again.

This is obviously not optimal. Let’s look at some solutions.

Solution 1: Start the Server in the Background

In most shells, you can run a command and instruct the shell to run it in the background. To do this, you end the line with an ampersand. Here’s how the example would look in this case:

By ending the command with an & ampersand symbol, we instruct the shell to run the command as a background job. This means that our shell is still functional. The shell has also notified us that this command is running as a background job with a specific job number:

% python -m SimpleHTTPServer 3000 &
[1] 19372

In slightly obtuse language, the shell has informed us that it has started a job in the background, with job number 1 and that this job is currently handling the process with ID 19372.

The ampersand solution is a fairly common pattern used in day-to-day work.

Solution 2: Move the Server to the Background

Let’s say you forgot to start the command in the background. Most likely in this case you’d kill the server with Ctrl+C and then start it again with the & option. However, what if this was a large file download or a task you didn’t want to abort?

In the example below, we’ll move the job to the background:

The process is currently in the foreground, so my shell is inactive. Hitting Ctrl+Z sends a ‘suspend’ signal to the process³, pausing it and moving it to the background.

Let’s dissect this:

% python -m SimpleHTTPServer 3000
Serving HTTP on 0.0.0.0 port 3000 ...
127.0.0.1 - - [03/Jun/2019 13:38:45] "GET / HTTP/1.1" 200 -
^Z
[1] + 21268 suspended python -m SimpleHTTPServer 3000

The shell echos as I type, so we see ^Z (i.e., the Ctrl+Z chord I entered). The shell responds by moving the process into a background job and suspending it.

The key here is that it is suspended. The process is paused. So the web server is no longer serving. If you are following with the sample, reload your browser. The webpage fails to load, as the server process is not able to respond to requests.

To continue the job, in the background, we use the bg (‘background’) command, with a job identifier (which always starts with a % symbol - we’ll see why soon) to tell the shell to continue the job:

% bg %1
[1] + 21268 continued python -m SimpleHTTPServer 3000

The shell lets us know the job is being continued, and if we load the webpage again, the content is shown as expected.

As a final check, we run the jobs command to see what jobs the shell is running:

% jobs
[1] + running python -m SimpleHTTPServer 3000

And there you have it - our server is running as a background job. This is exactly what we would see if we run jobs after starting the server with an & at the end. In fact, using an & is perhaps an easier way to remember how to continue a suspended job:

% %1 &
[1] + 21268 continued python -m SimpleHTTPServer 3000

In the same way ending a command with & runs it in the background, ending a job identifier with & continues it in the background.

There is at least one more way to move a job to the background⁴, but I have not yet found it useful in any scenarios, and it is overly complex to explain. See the footnote for details if you are interested.

Moving Background Jobs to the Foreground

If you have a job in the background, you can bring it back to the foreground with the fg (‘foreground’) command. Let’s show the jobs, with the jobs command:

% jobs
[1] + running python -m SimpleHTTPServer 3000

Here I have a background job running a server. Any one of the following commands will bring it back to the foreground:

fg %1 # Explicitly bring Job 1 into the foreground

%1 # ...or in shorthand, just enter the job id...

fg # ...if not given an id, fg and bg assume the most recent job.

Now the job is in the foreground, and you can interact with the process again however you like.

Cleaning Up Jobs

You might realise you cannot continue what you are doing because an old job is still running. Here’s an example:

I tried to run my web server, but there was still one running as a background job. The server failed to start because the port is in use.

To clean it up, I run the jobs command to list the jobs:

% jobs
[1] + suspended python -m SimpleHTTPServer 3000

There’s my old web server. Note that even though it is suspended, it’ll still be blocking the port it is serving on⁵. The process is paused, but it is still holding onto all of the resources it is using.

Now that I know the job identifier (%1 in this case), I can kill the job:

% kill %1
[1] + 22843 terminated python -m SimpleHTTPServer 3000

This is why job identifiers start with a percentage sign! The kill command I have used is not a special job control command (like bg or fg). It is the normal kill command, which terminates a process. But shells that support job control can normally use a job identifier in place of a process identifier. So rather than working out what the process identifier is that I need to kill, I can just use the job identifier⁶.

Why You Shouldn’t Use Jobs

Avoid jobs. They are not intuitive to interface with, and they suffer from some serious problems.

The most obvious one is that all jobs write to the same output, meaning you can quickly get garbled output like this:

This is what happens when I run a job, which just outputs text every second. It’s in the background, but it’s printing all over my commands. Even running the jobs command to try and find the job to stop it is difficult.

Input is even more complex. If a job is running in the background, but requires input, it will be silently suspended. This can cause confusion.

Jobs can be used in scripts but must be done so with caution and could easily confuse a consumer of the script if they leave background jobs hanging around, which cannot be easily cleaned up⁷.

Handling errors and exit codes for jobs can be problematic, causing confusion, poor error handling, or overly complex code.

How to Escape Jobs

If there are two things to take away, it would be this:

If you have started running a command in the foreground, and you don’t want to stop it and would rather move it to the background, hit Ctrl+Z. Then Google “job control”.

And:

If you think there is a job running in the background, and it is messing with your screen, type fg to bring it to the front and kill it with Ctrl+C. Repeat as needed!

In either case, if you need to do something more subtle, you can return to this reference. But the first command should allow you to get your shell back while you work out how to continue the job, and the second should kill a background job that is messing with your screen.

Alternatives to Jobs

If you are using any kind of modern terminal such as iTerm, Terminal or the GNOME Terminal, just open a new tab or split! Much easier.

The benefit to this is that each tab gets its own standard input and output, so there’s no risk of overwriting. And of course you can hide/reveal/rearrange the tabs however you like.

The traditional alternative to a job for an operator who simply wants more than one thing going on at once would be a terminal multiplexer, such as screen or tmux:

Multiplexers work in a very similar way to a modern graphical terminal - they manage many shell instances. The benefits to a modern terminal, such as iTerm, is that you have a very intuitive GUI and lots of features.

The benefits to a multiplexer are that you can run them over SSH sessions to manage complex operations on remote machines and that they run a client-server model, meaning many people can work with many multiplexed processes (and they can persist beyond sessions).

My personal preference is both - I use a modern terminal and run everything inside it in tmux. We’ll look at both of these options in later chapters.

Quick Reference

You might find that jobs are useful, or you might find that they are not. Either way, here’s a quick reference of some common commands:

Command	Usage
`command &`	Run the command as a background job.
`<Ctrl+Z>`	Move the current process into a background job, suspended.
`jobs`	List all jobs.
`fg %1`	Move background job number 1 into the foreground.
`bg %1`	Continue background job number 1.
`kill %1`	Terminate job number 1.
`wait %1`	Block until job number 1 exits.

If you want to find out more about the gory details of jobs, the best place to start is the Bash Manual - Job Control Section, or the ‘Job Control’ section of your preferred shell’s manual.

I hope you found this useful, and, as always, please leave comments, questions or suggestions below!

Footnotes

If you are not a heavy shell user, this might seem unlikely. But if you do a lot of work in shells, such as sysadmin, devops, or do your coding from a terminal, this happens all the time! ↩︎
Signals like SIGINT, SIGKILL, SIGTERM and so on will be covered in a later chapter. ↩︎
Technically, SIGTSTP - which is ‘TTY stop’. If you have always wondered about the ‘TTY’ acroynm, check the previous chatper, Interlude: Understanding the Shell. ↩︎
The alternative method is to use Ctrl+Y, which will send a delayed interrupt, which will continue to run the process until it tries to read from stdin. At this point, the job is suspended and the control given to the shell. The operator can then use bg or kill or fg to either move to the background, stop the process, or keep in the foreground as preferred. See: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Job-Control ↩︎
Another super-useful snippet: lsof -i -P -n | grep 8000 to find any process that has a given port open. Another one for the aliases chapter! ↩︎
There are times this is needed. If a job runs many processes - for example, by running a pipeline - the process identifier will change as the command moves from one stage of the pipeline to the next. The job identifier will remain constant. Remember, a job is a shell command, so could run many processes. ↩︎
To see how bad this can be, create a script that starts jobs, then run it. Then run the jobs command to see what is running. The output might surprise you! ↩︎

Effective Shell Interlude: Understanding the Shell

Tue, 21 May 2019 09:22:05 +0000

This is the first ‘interlude’ in my Effective Shell series. These interludes give some background, history or more flavour to some of the topics.

This one should be high-level enough for even non-technical readers to enjoy (or at least understand!). I’ve tried to make sure any term that might be unfamiliar is described in a footnote¹. For the more technical reader, it provides an important grounding on some of the key concepts relating to shells and how they work.

Introduction for the Non-Technical Reader

It might come as a surprise that many technical computer users (programmers, data scientists, systems administrators etc) spend a lot of time using an interface which looks like it’s from the sixties:

If you work with technologists, you might have seen them using an interface like this. This kind of simple, text-based interface is called a shell, and it has been a common way to interface with computers ever since the first screens and keyboards were created.

Given how much computing has advanced, why would people use such an interface? Just look at how much the Windows operating-system has changed over the last three decades:

(By Source (WP:NFCC#4), Fair use, https://en.wikipedia.org/w/index.php?curid=58853841)

Why would people choose to use such an archaic interface as a shell?

Typing is fast: A skilled shell user can manipulate a system at dazzling speeds just using a keyboard. Typing commands is generally much faster than exploring through user interfaces with a mouse
Shells are programmable: Users will often being programming as they work in a shell, creating scripts to automate time-consuming or repetetive processes
Shells are portable: A shell can be used to interface to almost any type of computer, from a mainframe to a Raspberry Pi, in a very similar way.

Not all technical users will use a shell regularly, but there are many who will spend the bulk of their time in such an interface. It is such a crucial skill to be able to operate one effectively that I have been writing this series primarily to show ways to be more efficient with this kind of interface.

Introduction for the Technical Reader

You may be familar with the shell, but it can be useful to understand some of the surrounding concepts in detail. How does a shell differ from a terminal? What is a tty? How do shells really work? Hopefully as you read this article you’ll discovery something that you didn’t know about shells.

Let’s Get Started!

To understand what shells, terminals, command-prompts and so on are and how they relate, we need to start with the basics: how a modern computer works!

A Computer in a Nutshell

The diagram below shows a simplified view of a typical computer:

Already there’s a lot going on.

Your computer is going to have a CPU² and memory³, and almost certainly a network adapter⁴ and display adapter⁵. Most computers will have at least one hard disk. For home PCs, there’ll also likely be a bunch of peripherals, such as a mouse, keyboard, printers, flash drives, webcams and so on.

The Operating System

The operating system is the piece of software installed on a computer that can interface with the hardware. Without hardware, such as a CPU, memory, a network adapter, a graphics card, disk drives and so on, there’s not much that you can do with the computer. The operating system is the primary interface to this hardware. No normal programs will talk to hardware directly - the operating system abstracts this hardware away and provides a software interface to it.

The abstraction the operating system provides is essential. Developers don’t need to know the specifics of how to work with individual devices from different vendors; the operating system provides a standardised interface to all of this. It also handles various tasks such as making sure the system starts up properly.

The operating system is generally broken down into two parts - the kernel and user space:

Let’s look at these in more detail.

The Kernel

This is the part of the operating system that is responsible for the most sensitive tasks: interfacing with physical devices, managing the resources that are available for users and programs, starting up the various systems that are needed, and so on.

Software running in the kernel has direct access to resources, so is extremely sensitive. The kernel will balance resources between the programs in user space, which we’ll look at shortly. If you’ve ever had to install ‘drivers’, these are examples of pieces of software that will run in the kernel - they’ll have direct access to a physical device you’ve installed, and expose it to the rest of the software on the computer.

Why ‘kernel’? The kernel is the soft, edible part of a nut or seed, which is surrounded by a shell. Below you can see a walnut - the kernel is the soft bit in the middle, and the shell surrounds and protects it. This is a useful metaphor that is used for parts of a computer.

(By Kkchaudhary11 - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=49069244)

The operating system kernel really is the core of the operating system. It’s such a sensitive area of the operating system that we actually want to avoid running software in it if possible⁶. And that is where user space comes in.

User Space

The vast majority of programs run in ‘user space’ (also commonly called ‘user land’).

When a program starts, the kernel will allocate it a private segment of memory and provide limited access to resources. The program is given access to a library of functions by the operating system, which it can use to access resources such as files, devices and so on. Programs in user space are essentially in sandboxes, where there is a limit to how much damage they can do.

For example, a program running in user space can use the standard fopen function, which is provided on almost every operating system as part of the C Standard Library. This allows a program to attempt to open a file. The operating system will make a decision on whether the program is allowed to open the file (based on things such as permissions, where the file is and so on) and then, if it is OK with the call, will give the program access to the file. Under the hood, this ‘user space’ call translates to a system call in the kernel.

Now that the key components have been introduced, we can look at the shell. The name should come as no surprise, as it is a wrapper or outer layer to the operating system (which itself contains the sensitive nugget of the kernel).

The Shell

So what is the shell? The shell is just a general name for any user space program that allows access to resources in the system, via some kind of interface.

Shells come in many different flavours but are generally provided to aid a human operator in accessing the system. This could be interactively, by typing at a terminal, or via scripts, which are files that contain a sequence of commands.

For example, to see all of the files in a folder, the human operator could write a program in a language such as C, making system calls to do what they want. But for day-to-day tasks, this would be repetitive. A shell will normally offer us a quick way to do that exact task, without having to manually write a program to do it.

Here’s an example, where a shell is being used to show the ‘png’ images in the folder I am working in⁷:

So a shell is a user-space program to interface with the computer. But there a few more moving parts than just a shell we are seeing in the image above. There are different types of shells, there are terminal programs, and there are the programs or commands that the shell calls (in the example above, tree is a program). Let’s pick these apart.

Here’s a diagram that more accurately shows what is going on:

We’ve introduced a few new things here. There’s a user, who is interfacing with a terminal, which is running a shell, which is showing a command prompt. The user has written a command that is calling a program (in this case, the tree program).

Let’s dissect this bit by bit.

The Terminal

We’re not directly interacting with the ‘shell’ in this diagram. We’re actually using a terminal. When a user wants to work with a shell interactively, using a keyboard to provide input and a display to see the output on the screen, the user uses a terminal.

A terminal is just a program that reads input from the keyboard, passes that input to another program (normally a shell), and displays the results on the screen. A shell program on its own does not do this - it requires a terminal as an interface.

Why the word terminal? This makes sense when you look at how people interfaced with computers historically. Input to a computer might be through punch cards, and output would often be via a printer. The Teletype Termimal⁸ became a common way for users to interface with computers.

(Photograph by Rama, Wikimedia Commons, Cc-by-sa-2.0-fr, CC BY-SA 2.0 fr, https://commons.wikimedia.org/w/index.php?curid=17821795)

At this time, computers were very large, complex, and expensive machines. It was common to have many terminals connected to a single large machine (or ‘mainframe’), or a few terminals that people would share. But the terminal itself was just a human interface to the operating system. A more modern terminal would be something like an IBM 3486:

(By ClickRick - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=6693700)

This is a very small computer in its own right but still basically just a dumb screen and keyboard connected by a cable to a larger mainframe computer in another location.

This mechanism is still very much the case today. When I want to work with a computer in a data centre, I don’t go and find the machine, plug in a keyboard and a display and directly interface to it. I run a terminal program on my computer to provide access to the remote machine. My terminal program allows me to use my keyboard and display to work with a remote machine - all via a secure shell - which is a secured-shell connection over a network.

So terminals in many ways are quite simple - they are interfaces. But because they are quite simple programs, we can’t do much with them. So normally, the first thing that a terminal program will do is run a shell program - a program that we can use to operate the computer.

There’s nothing special about terminals - anyone can write a program to operate as a terminal, which is why you will see many different terminals around. Examples are the standard ’terminal’ app for MacOS X, the gnome-terminal for Linux, and iTerm2 and Hyper. There’s a bunch of screenshots of different setups at the end of the article.

Back to the Shell

Now that we’ve described the terminal, we can go back and look at the shell in detail.

The shell is the program that is going to take input from somewhere and run a series of commands. When the shell is running in a terminal, it is normally taking input interactively from the user. As the user types in commands, the terminal feeds the input to the shell and presents the output of the shell on the screen.

A shell program can also take input from files; these files will then generally be ‘shell scripts’. This might be used to run automated operations, such as cleaning up certain folders when a computer starts.

Shells can write output to files or other locations, and so on. You can run a shell program outside of a terminal - you just won’t be able to interface with it using a keyboard or display. And in fact, lots of operations happen in this way: automated scripts, startup tasks, installers and so on.

So what else does a shell do? Most of the features are related to helping human operators work with the system more efficiently.

Quickly enter commands, see the history of commands and quickly restructure commands (see Effective Shell - Navigating the Command Line)
Navigate through the file system, moving from folder to folder (see Effective Shell - Move Around!), which makes it easier for an operator to navigate the file system.
Chain the output of commands together - for example, taking the output of one basic program, such as the tree program we saw, and writing it to a file (see Effective Shell - Understanding Pipelines)
Offer a programming language, allowing the operator to perform more complicated tasks (see Effective Shell - Basic Shell Scripting)

And a lot more! In fact, that’s what the whole Effective Shell series is about - how to get the most from these powerful programs, particularly for those who use them regularly.

The Command Prompt or Command Line

The last part of the diagram, which we haven’t covered yet, is the command prompt.

When a shell is running in terminal, it knows that a human operator will be interfacing with it. So to make sure that the operator has some kind of visual hint that they have to enter commands, the shell will output some kind of prompt.

I’ve included a set of screenshots at the end of the article, just after this section, and you can see how some different command prompts look.

Note that shells don’t have to use command prompts - if you use a shell program to execute a script, there will be no command prompt. Shells only show a prompt when they know they are being used interactively. Many programs which allow a user to operate interactively will show a command prompt.

Shell command prompts can be customised, so they will often look different from machine to machine (for more details, see Effective Shell - Customising the Command Line). Below is an example that shows a lot of technical information. This is from the highly popular oh-my-zsh framework for the ‘Z Shell’ shell, which is very popular among developers:

*(Source: https://ohmyz.sh/)

Shell Commands and Different Shells

A lot of the ‘commands’ in a shell, such as cat (which shows the contents of a file), are actually just simple programs, which will interface with the kernel. No matter what shell you use, these commands will behave the same way, because really all you are doing is calling another progam.

Some commands, such as cd (change directory), are built into the shell. Some commands are functions that have been defined, or aliases to other commands (for more details on commands, see Effective Shell - Commands). Commands will often differ between shells.

Not all shells are created equal - anyone can write a shell program, maybe creating a simple interface to the computer or a highly complex one with many features. In fact, a later article in this series will look at the geneology of the most common shells.

On most Unix-like systems, the default shell is a program called bash, which stands for " Bourne Again Shell" (the name and history around it will be discussed at length in the later article). But there are many other shells: the C Shell, the Korn Shell, Z Shell and Fish, just to name just a few.

Users and administators can configure what shell they like to use. When a terminal opens, it will immediately start the user’s preferred shell program. It is possible to change this. Different users will have different preferences, given that shells offer varying features. This can cause complexity when working with systems, as we cannot always expect every user to have the same shell, or even for the same shell to be set up consistently, as they can be extensively customised.

Let’s review the earlier diagram again:

We can see the real internals of what is going on in this “Terminal -> Shell -> Program” chain in the diagram above quite easily.

Try the command pstree -psa $$ in a shell⁹:

The first systemd process is the primary process for the OS - it is process number 1, which initialises everything else. The second systemd process is the process that is running the interface for my user. We can ignore these for now; they are internals to how the operating system boots and starts processes.

What is interesting is that we can see a terminal (the gnome terminal), which has started my preferred shell (which is zsh), which is running a command (the program pstree). Here we can see the exact chain as shown in the diagram earlier.

That’s a Wrap!

These are the key technologies and concepts that surround a shell.

If you are interested in more technical details of working with shells, then my Effective Shell series goes into these topics in depth. The goal of this series is to help teach techniques that making working with shells more efficient.

To close the article, below are some examples of different terminals, shells, command prompts and so on.

Example: iTerm 2 / tmux / zsh

In this example, we have:

A MacOS operating system
iTerm2 as the terminal program
tmux running as a ’terminal multiplexer’ (see Effective Shell: Terminal Multiplexers)
zsh (Z Shell) as the shell program, using ‘oh my zsh’, which is easily recognised by the % sign in the command prompt.
A customised command line, which shows the user and folder on one line, with only the % symbol below, to leave lots of space for the input commands¹⁰.

Example: Bash

In this example, we have:

A Linux operating system (Ubuntu 14)
The gnome terminal
bash as the shell
In the second screenshot, the user has ‘root privileges’, and to indicate this, bash helpfully changes the default command prompt from a dollar sign to a hash sign

Example: Windows Explorer

In this example, we have:

The Windows 10 operating system
No terminal
The explorer.exe program showing us a graphical shell

This looks different from previous examples. The program, which shows the familiar Windows interface, explorer.exe, is in fact a shell as well, offering interactive access to the operating system and computer resources. The bulk of the Windows APIs to interact with this interface are in the Shell Library. I also maintain a popular library for building extensions to the graphical Windows shell - sharpshell.

Example: Windows Command Prompt

In this example, we have:

The Windows 10 operating system
The command prompt terminal and shell

In Windows, the terminal and shell are combined into a single cmd.exe program. There’s an excellent article on the internals - Microsoft DevBlogs: Windows Command-Line: Inside the Windows Console

Example: Windows PowerShell

In this example, we have:

The Windows 10 operating system
The PowerShell terminal

PowerShell is an improvement on the ‘command prompt’ program that was originally used in Windows, offering much more functionality for scripting and other modern shell features.

Example: Windows Subsystem for Linux (WSL)

In this example, we have:

The Windows 10 operating system
The Bash.exe program

This screenshot, from MSDN: Frequently Asked Questions about Windows Subsystem for Linux shows Bash running in Windows. This is a relatively new feature at the time of writing, allowing Windows users to use a Linux interface to the PC. This is a feature that may become increasingly valuable, as in general it is challenging to write shell code that can run on Windows and Unix-like systems.

If you enjoyed this article, please do share it! Feel free to include suggestions, improvements or corrections in the comments below.

Useful References

A simple Linux kernel module, showing how basic kernel programming works in Linux: github.com/dwmkerr/linux-kernel-module
How Linux Works - Brian Ward
StackExchange: What is the exact difference between a ’terminal’, a ‘shell’, a ’tty’, and a console?
Microsoft: Inside the Windows Console

Footnotes

I’d be fascinated to know if this is at all interesting to less technically inclined people, so please do go ahead and let me know in the comments! ↩︎
CPU: central processing unit. This is the chip in the computer that does most of the work (which after many layers of abstraction eventually becomes arithmetic and sending simple instructions to other places). ↩︎
Memory is the ‘working space’ where the state of your system is stored. If you are writing a document, the text lives in memory, until you save it, when it then gets written to a hard drive. Memory is ephemeral - everything is gone when you turn off the power to it. ↩︎
This is the part of your computer that knows how to do things like connect to a WiFi network, or has a network socket you might plug a network cable into. ↩︎
This is the part of your computer you plug the screen into. ↩︎
This is because a mistake in Kernel Mode programs can have disasterous effects. It could access any files, no matter who they belong do, control the hardware, install more software - almost anything. Errors in this code can cause terrible issues (like the infamous Windows ‘blue screen of death’), and malicious code in the kernel essentially has full access to not only all your data but also your webcam, network adapter and so on. ↩︎
As an aside, if you are curious about the visual style of my setup or customisations that have been made, everything in my setup is available online on my ‘dotfiles’ repo - github.com/dwmkerr/dotfiles. ↩︎
And that’s where the ‘TTY’ acronym you will see sometimes comes from. Enter the ps command, and you’ll actually see the TTY interface each process is attached to. This is a topic that will come up later in the series. ↩︎
$$ is a Bash internal variable. These will also be covered in a later article in the series. ↩︎
Feel free to see my dotfiles to configure a similar setup for yourself. ↩︎

Manipulating Istio and other Custom Kubernetes Resources in Golang

Mon, 08 Oct 2018 21:34:02 +0000

In this article I’ll demonstrate how to use Golang to manipulate Kubernetes Custom Resources, with Istio as an example. No knowledge of Istio is needed, I’ll just use it to demonstrate the concepts!

Istio is a highly popular Service Mesh platform which allows engineers to quickly add telemetry, advanced traffic management and more to their service-based applications.

One interesting element of how Istio works is that when deployed into a Kubernetes cluster, many key configuration objects are handled as Custom Resources. Custom Resources are a very powerful Kubernetes feature, which allow you to create your own ‘first class’ resources (just like Pods, ReplicaSets, Deployments or whatever) and then interface with them using kubectl or the Kubernetes APIs.

In this article I’ll show you how to interface with these Custom Resources using the Golang Kubernetes client.

CRDs: A Quick Overview

When you set up Istio for your cluster, one common thing you will likely do is specify how you will route traffic. This can be quite sophisticated, as shown below:

Figure 1: Istio Traffic Management Examples, from istio.io

One way for a system like this to be configured would be to have a ConfigMap which contains the definition of how services are routed.

However, Istio actually registers new types of resources (Custom Resource Definitions) which represent things like Gateways or Services. We can create/update/delete/manipulate them just like any other Kubernetes object.

For example, I could create a virtual service for the example above with something like this:

cat << EOF | kubectl create -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: service2
spec:
 hosts:
 - "*"
 gateways:
 - demo1-gateway
 http:
 - route:
 - destination:
 host: service2
 subset: v1
 weight: 95
 - destination:
 host: service2
 subset: v2
 weight: 5
EOF

Again, the important thing is not the specific content of this resource, more the fact that I can treat my Istio resources just like I would any other Kubernetes object:

$ kubectl get virtualservices.networking.istio.io
NAME AGE
service2 93s

Or:

$ kubectl delete virtualservices.networking.istio.io/service2

I can use edit, describe, register lifecycle events, watch for changes, and so on.

Working with CRDs in Golang

The Golang Kubernetes Client allows you to create strongly defined types which you can then use to interface with CRDs. An example is in the Red Hat blog post Kubernetes Deep Dive: Code Generation for Custom Resources.

This is an excellent approach, but can feel pretty heavy if you want to quickly access some data, and don’t want to have to generate a lot of code.

There is an alternative, which is to use the DynamicClient. The preferred approach seems to be the first, which involves code generation, so little documentation exists for the second approach. However, it is actually very simple.

Here’s an example of how you can list all Istio VirtualService resources, without having to generate any code:

import (
 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/dynamic"
)

// Create a Dynamic Client to interface with CRDs.
dynamicClient, _ := dynamic.NewForConfig(config)

// Create a GVR which represents an Istio Virtual Service.
virtualServiceGVR := schema.GroupVersionResource{
 Group: "networking.istio.io",
 Version: "v1alpha3",
 Resource: "virtualservices",
}

// List all of the Virtual Services.
virtualServices, _ := dynamicClient.Resource(virtualServiceGVR).Namespace("default").List(metav1.ListOptions{})
for _, virtualService := range virtualServices.Items {
 fmt.Printf("VirtualService: %s\n", virtualService.GetName())
}

This snippet omits setup and error-handling for clarity, the full example is in the k8s-list-virtualservices.go gist.

Patching CRDs in Golang

You may have noticed that the .Resource().Namespace().List() code looks very similar to the structure for making API calls when using the Kubernetes Clientset. In fact, it is essentially the same. Looking at the interface, you can see you have all of the operations you’d expect:

Create
Update
Delete
Get

And so on. This is nice because you can use the same trick in my article ‘Patching Kubernetes Resources in Golang’ to manipulate these entities, without ever having to create a structure to represent it.

Here’s another abbreviated example, this time showing how we can adjust the weight of the routing from the services to 50%/50%:

import (
 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/dynamic"
)

// Create a GVR which represents an Istio Virtual Service.
virtualServiceGVR := schema.GroupVersionResource{
 Group: "networking.istio.io",
 Version: "v1alpha3",
 Resource: "virtualservices",
}

// Weight the two routes - 50/50.
patchPayload := make([]PatchUInt32Value, 2)
patchPayload[0].Op = "replace"
patchPayload[0].Path = "/spec/http/0/route/0/weight"
patchPayload[0].Value = 50
patchPayload[1].Op = "replace"
patchPayload[1].Path = "/spec/http/0/route/1/weight"
patchPayload[1].Value = 50
patchBytes, _ := json.Marshal(patchPayload)

// Apply the patch to the 'service2' service.
_, err := dynamicClient.Resource(virtualServiceGVR).Namespace("default").Patch("service2", types.JSONPatchType, patchBytes)

See the full example in the gist k8s-patch-virtualservice.go

After running the sample, you can use the Kubernetes CLI to verify the changes:

$ kubectl get virtualservices.networking.istio.io/service2 -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
clusterName: ""
creationTimestamp: 2018-10-08T09:53:16Z
generation: 0
name: service2
namespace: default
resourceVersion: "487435"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/service2
uid: fac5930c-cadf-11e8-90a2-42010a94005b
spec:
gateways:
- demo1-gateway
hosts:
- '*'
http:
- route:
- destination:
host: service2
subset: v1
weight: 50
- destination:
host: service2
subset: v2
weight: 50

Keep It Simple!

That’s it! This trick made something I was working on a lot easier, but it took a little bit of experimentation to get right. I hope you find the approach useful. Please share any thoughts/questions in the comments.

Patching Kubernetes Resources in Golang

Tue, 24 Jul 2018 06:33:17 +0000

Recently I needed to be able to quickly adjust the number of replicas in a Kubernetes Replication Controller. The original solution I’d seen pulled down the spec, modified it, then updated it. There’s a better way!

There’s a patch API for Kubernetes resources. Patching resources is faster and easier than pulling them and updating the spec wholesale. However, the documentation is a little limited.

After some trial and error I got it working, here’s the solution. I thought it might be helpful to share for others!

The Solution

I’ll start with the solution. If this is all you need, you are good to go. The details of how this works are presented afterwards. In this example I’ll update the number of replicas in the my-rc controller:

package main

import (
 "encoding/json"
 "fmt"

 types "k8s.io/apimachinery/pkg/types"
 "k8s.io/client-go/kubernetes"
 _ "k8s.io/client-go/plugin/pkg/client/auth"
 "k8s.io/client-go/tools/clientcmd"
)

var (
 // Leave blank for the default context in your kube config.
 context = ""

 // Name of the replication controller to scale, and the desired number of replicas.
 replicationControllerName = "my-rc"
 replicas = uint32(3)
)

// patchStringValue specifies a patch operation for a string.
type patchStringValue struct {
 Op string `json:"op"`
 Path string `json:"path"`
 Value string `json:"value"`
}

// patchStringValue specifies a patch operation for a uint32.
type patchUInt32Value struct {
 Op string `json:"op"`
 Path string `json:"path"`
 Value uint32 `json:"value"`
}

func scaleReplicationController(clientSet *kubernetes.Clientset, replicasetName string, scale uint32) error {
 payload := []patchUInt32Value{{
 Op: "replace",
 Path: "/spec/replicas",
 Value: scale,
 }}
 payloadBytes, _ := json.Marshal(payload)
 _, err := clientSet.
 CoreV1().
 ReplicationControllers("default").
 Patch(replicasetName, types.JSONPatchType, payloadBytes)
 return err
}

func main() {
 // Get the local kube config.
 fmt.Printf("Connecting to Kubernetes Context %v\n", context)
 config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
 clientcmd.NewDefaultClientConfigLoadingRules(),
 &clientcmd.ConfigOverrides{CurrentContext: context}).ClientConfig()
 if err != nil {
 panic(err.Error())
 }

 // Creates the clientset
 clientset, err := kubernetes.NewForConfig(config)
 if err != nil {
 panic(err.Error())
 }

 // Scale our replication controller.
 fmt.Printf("Scaling replication controller %v to %v\n", replicationControllerName, replicas)
 err = scaleReplicationController(clientset, replicationControllerName, replicas)
 if err != nil {
 panic(err.Error())
 }
}

This code is also available in the k8s-patch.go gist.

The Mechanism

The Kubernetes Patch API supports a few different methods for modifying resources. It is important to be aware that there is not a universally accepted ‘standard’ approach to representing a change to a resource in a REST API.

There are three strategies you can use to patch:

merge: follows the JSON Merge Patch Spec (RFC 7386)
stragetic: A strategic merge, which addresses some limitations of the merge patch (noted in this doc.
json: follows the JSON Patch Spec (RFC 6902)

These are documented in detail at:

docs/devel/api-conventions.md#patch-operations

The mechanism I’ve used here is json, which I think is the clearest to the reader. To use this strategy we need to build a payload describing what we are changing. This might look like this:

{
 "op": "replace",
 "path": "/spec/replicas",
 "value": 4
}

The op field can be remove, replace, add etc etc (all the details are in the RFC 6902), or the slightly more readable jsonpatch.com). This allows the operation to be very explicit to the reader, which is helpful. We create a struct which represents an operation on a string or integer (or whatever data type we need), serialize it and pass to the API.

Under the hood, the Golang client will simply translate this into an HTTP call which will look like something like this:

PATCH /api/v1/namespaces/default/replicationcontrollers/app-server-blue HTTP/1.1
Host: 127.0.0.1
Content-Type: application/json-patch+json
Content-Length: 70
[{
"op": "replace",
"path": "/spec/replicas",
"value": 4
}]

This corresponds to the documentation on the Patch Operations. Note that the patch operation type is specified in the Content-Type header.

Hopefully this’ll help you if you need to patch resources, are struggling with the docs and are a Go noob like me! Any tips on how to make the code cleaner or more idomatic would be welcome.

Thanks to the following articles and issues which helped me unpick this:

mongo-monitor - a simple CLI to monitor your MongoDB cluster

Wed, 16 May 2018 20:09:53 +0000

The mongo-monitor CLI is a lean and simple tool to check the status of a MongoDB server or cluster. The code is on GitHub:

github.com/dwmkerr/mongo-monitor

Here’s how it looks in action:

In this animation I am monitoring a simple sharded cluster, and running some example maintenance operations, adding a node to a replicaset, stepping down a primary and shutting down a replicaset node.

A simple CLI which shows the status in real-time can be very useful to keep open when performing admin, letting you see how your changes affect the cluster as you work on it.

Installing the CLI

The CLI is installed with npm:

npm install -g mongo-monitor

Connecting to a Cluster

Connect to a cluster by providing a connection string. The tool uses mongo-connection-string to parse the connection string, so you can be flexible with the input:

# Connect to a local instance
mongo-monitor localhost:27107

# Connect to a remote replicaset, authenticated
mongo-monitor admin:P@sswrd@mdbnode1,mdbnode2,mdbnode3?replicaSet=rs

Once a connection is established, the tool will periodically check the status of the cluster. If the cluster is sharded, it will also inspect each individual replicaset.

Replicaset Status

Here’s the kind of output you might get from a replicaset:

The name of the replicaset is shown, along with each member. The status of each member is also shown, updating automatically every second.

This is convenient when administering replicasets, stepping down a master, adding or removing nodes and so on.

Sharded Cluster Status

When connecting to a sharded cluster, you will get output like this:

Each shard is shown, along with the details of the replicaset which make it up.

Keeping a view like this open is useful when administering sharded clusters, adding or removing shards, desharding, updating the replicasets which make up shards and so on.

Get Involved!

If you like the tool, check out the code and feel free to make pull requests with additions! There are a few issues on the project already, and there are all sorts of features I’d love to add but haven’t had the time, such as:

Being able to see the lag for replicaset members, to see if secondaries are falling behind
Being able to perform replicaset operations directly from the tool
Showing the priorities of nodes if they are not the default

All ideas are welcome, let me know in the comments or repo, and share the tool if you find it useful!

The Death of Microservice Madness in 2018

Fri, 12 Jan 2018 10:52:25 +0000

En Español | Reddit Thread | Hacker News Thread

Microservices became a very popular topic over the last couple of years¹. ‘Microservice madness’ goes something like this:

Netflix are great at devops. Netflix do microservices. Therefore: If I do microservices, I am great at devops.

There are many cases where great efforts have been made to adopt microservice patterns without necessarily understanding how the costs and benefits will apply to the specifics of the problem at hand.

I’m going to describe in detail what microservices are, why the pattern is so appealing, and also some of the key challenges that they present.

I’ll finish with a set of simple questions might be valuable to ask yourself when you are considering whether microservices are the right pattern for you. The questions are at the end of the article.

What are microservices, and why are they so popular?

Let’s start with the basics. Here is how a hypothetical video sharing platform might be implemented, first in the form of a monolith (single large unit) and then in the form of microservices:

The difference between the two systems is that the first is a single large unit; a monolith. The second is a set of small, specific services. Each service has a specific role.

When the diagram is drawn at this level of detail, it is easy to see the appeal. There are a whole host of potential benefits:

Independent Development: Small, independent components can be built by small, independent teams. A group can work on a change to the ‘Upload’ service without interfering with the ‘Transcode’ service, or even knowing about it. The amount of time to learn about a component is greatly reduced, and it is easier to develop new features.

Independent Deployment: Each individual component can be deployed independently. This allows new features to be released with greater velocity and less risk. Fixes or features for the ‘Streaming’ component can be deployed without requiring other components to be deployed.

Independent Scalability: Each component can be scaled independently of each other. During busy periods when new shows are released, the ‘Download’ component can be scaled up to handle the increased load, without having to scale up every component, which makes elastic scaling more feasible and reduces costs.

Reusability: Components fulfil a small, specific function. This means that they can more easily be adapted for use in other systems, services or products. The ‘Transcode’ component could be used by other business units, or even turned into a new business, perhaps offering transcoding services for other groups.

At this level of detail, the benefits of a microservice model over a monolithic model seem obvious. So if that’s the case - why is this pattern only recently in vogue? Where has it been all my life?

If this is so great, why hasn’t it been done before?

There are two answers to this question. One is that it has - to the best of our technical capabilities, and the other is that more recent technical advances have allowed us to take it to a new level.

When I started writing the answer to this question, it turned into a long description, so I’m actually going to separate it into another article and publish it a little later². At this stage, I will skip the journey from single program to many programs, ignore ESBs and Service Orientated Architecture, component design and bounded contexts, and so on.

Those who are interested can read more about the journey separately. Instead I’ll say that in many ways we’ve been doing this for a while, but with the recent explosion in popularity of container technology (Docker in particular) and in orchestration technology (such as Kubernetes, Mesos, Consul and so on) this pattern has become much more viable to implement from a technical standpoint.

So if we take it as a given that we can implement a microservice arrangement, we need to think carefully about the should. We’ve seen the high-level theoretical benefits, but what about the challenges?

What’s the problem with microservices?

If microservices are so great, what’s the big deal? Here are some of the biggest issues I’ve seen.

Increased complexity for developers

Things can get a lot harder for developers. In the case where a developer wants to work on a journey, or feature which might span many services, that developer has to run them all on their machine, or connect to them. This is often more complex than simply running a single program.

This challenge can be partially mitigated with tooling³, but as the number of services which makes up a system increases, the more challenges developers will face when running the system as a whole.

Increased complexity for operators

For teams who don’t develop services, but maintain them, there is an explosion in potential complexity. Instead of perhaps managing a few running services, they are managing dozens, hundreds or thousands of running services. There are more services, more communication paths, and more areas of potential failure.

Increased complexity for devops

Reading the two points above, it may grate that operations and development are treated separately, especially given the popularity of devops as a practice (which I am a big proponent of). Doesn’t devops mitigate this?

The challenge is that many organisations still run with separated development and operations teams - and a organisation that does is much more likely to struggle with adoption of microservices.

For organisations which have adopted devops, it’s still hard. Being both a developer and an operator is already tough (but critical to build good software), but having to also understand the nuances of container orchestration systems, particularly systems which are evolving at a rapid pace, is very hard. Which brings me onto the next point.

It requires serious expertise

When done by experts, the results can be wonderful. But imagine an organisation where perhaps things are not running smoothly with a single monolithic system. What possible reason would there be that things would be any better by increasing the number of systems, which increases the operational complexity?

Yes, with effective automation, monitoring, orchestration and so on, this is all possible. But the challenge is rarely the technology - the challenge is finding people who can use it effectively. These skillsets are currently in very high demand, and may be difficult to find.

Real world systems often have poorly defined boundaries

In all of the examples we used to describe the benefits of microservices, we spoke about independent components. However in many cases components are simply not independent. On paper, certain domains may look bounded, but as you get into the muddy details, you may find that they are more challenging to model than you anticipated.

This is where things can get extremely complex. If your boundaries are actually not well defined, then what happens is that even though theoretically services can be deployed in isolation, you find that due to the inter-dependencies between services, you have to deploy sets of services as a group.

This then means that you need to manage coherent versions of services which are proven and tested when working together, you don’t actually have an independently deployable system, because to deploy a new feature, you need to carefully orchestrate the simultaneous deployment of many services.

The complexities of state are often ignored

In the previous example, I mentioned that a feature deployment may require the simultaneous rollout of many versions of many services in tandem. It is tempting to assume that sensible deployment techniques will mitigate this, for example blue/green deployments (which most service orchestration platforms handle with little effort), or multiple versions of a service being run in parallel, with consuming channels deciding which version to use.

These techniques mitigate a large number of the challenges if the services are stateless. But stateless services are quite frankly, easy to deal with. In fact, if you have stateless services, then I’d be inclined to consider skipping microservices altogether and consider using a serverless model.

In reality, many services require state. An example from our video sharing platform might be the subscription service. A new version of the subscriptions service may store data in the subscriptions database in a different shape. If you are running both services in parallel, you are running the system with two schemas at once. If you do a blue green deployment, and other services depend on data in the new shape, then they must be updated at the same time, and if the subscription service deployment fails and rolls back, they might need to roll back too, with cascading consequences.

Again, it might be tempting to think that with NoSQL databases these issues of schema go away, but they don’t. Databases which don’t enforce schema do not lead to schemaless systems - they just mean that schema tends to be managed at the application level, rather than the database level. The fundamental challenge of understanding the shape of your data, and how it evolves, cannot be eliminated.

The complexitities of communication are often ignored

As you build a large network of services which depend on each other, the liklihood is that there will be a lot of inter-service communication. This leads to a few challenges. Firstly, there are a lot more points at which things can fail. We must expect that network calls will fail, which means when one service calls another, it should expect to have to retry a number of times at the least. Now when a service has to potentially call many services, we end up in a complicated situation.

Imagine a user uploads a video in the video sharing service. We might need to run the upload service, pass data to the transcode service, update subscriptions, update recommendations and so on. All of these calls require a degree of orchestration, if things fail we need to retry.

This retry logic can get hard to manage. Trying to do things synchronously often ends up being untenable, there are too many points of failure. In this case, a more reliable solution is to use asynchronous patterns to handle communication. The challenge here is that asynchronous patterns inherently make a system stateful. As mentioned in the previous point, stateful systems and systems with distributed state are very hard to handle.

When a microservice system uses message queues for intra-service communication, you essentially have a large database (the message queue or broker) glueing the services together. Again, although it might not seem like a challenge at first, schema will come back to bite you. A service at version X might write a message with a certain format, services which depend on this message will also need to be updated when the sending service changes the details of the message it sends.

It is possible to have services which can handle messages in many different formats, but this is hard to manage. Now when deploying new versions of services, you will have times where two different versions of a service may be trying to process messages from the same queue, perhaps even messages sent by different versions of a sending service. This can lead to complicated edge cases. To avoid these edge cases, it may be easier to only allow certain versions of messages to exist, meaning that you need to deploy a set of versions of a set of services as a coherent whole, ensuring messages of older versions are drained appropriately first.

This highlights again that the idea of independent deployments may not hold as expected when you get into the details.

Versioning can be hard

To mitigate the challenges mentioned previously, versioning needs to be very carefully managed. Again, there can be a tendency to assume that following a standard such as semver[4] will solve the problem. It doesn’t. Semver is a sensible convention to use, but you will still have to track the versions of services and APIs which can work together.

This can get very challenging very quickly, and may get to the point where you don’t know which versions of services will actually work properly together.

Managing dependencies in software systems is notoriously hard, whether it is node modules, Java modules, C libraries or whatever. The challenges of conflicts between independent components when consumed by a single entity are very hard to deal with.

These challenges are hard to deal with when the dependencies are static, and can be patched, updated, edited and so on, but if the dependencies are themselves live services, then you may not be able to just update them - you may have to run many versions (with the challenges already described) or bring down the system until it is fixed holistically.

Distributed Transactions

In situations where you need transaction integrity across an operation, microservices can be very painful. Distributed state is hard to deal with, many small units which can fail make orchestrating transactions very hard.

It may be tempting to attempt to avoid the problem by making operations idempotent, offering retry mechanisms and so on, and in many cases this might work. But you may have scenarios where you simply need a transaction to fail or succeed, and never be in an intermediate state. The effort involved in working around this or implementing it in a microservice model may be very high.

Microservices can be monoliths in disguise

Yes, individual services and components may be deployed in isolation, however in most cases you are going to have to be running some kind of orchestration platform, such as Kubernetes. If you are using a managed service, such as Google’s GKE⁴ or Amazon’s EKS⁵, then a large amount of the complexity of managing the cluster is handled for you.

However, if you are managing the cluster yourself, you are managing a large, complicated, mission critical system. Although the individual services may have all of the benefits described earlier, you need to very carefully manage your cluster. Deployments of this system can be hard, updates can be hard, failover can be hard and so on.

In many cases the overall benefits are still there, but it is important not to trivialise or underestimate the additional complexity of managing another big, complex system. Managed services may help, but in many cases these services are nascent (Amazon EKS was only announced at the end of 2017 for example).

Networking Nightmares

A more traditional model of services running on known hosts, with known addresses, has a fairly simple networking setup.

However, when using microservices, generally there will be many services distributed across many nodes, which typically means there’s going to be a much more complicated networking arrangement. There will be load balancing between services, DNS may be more heavily used, virtual networking layers, etc etc, to attempt to ‘hide’ the complexity of this networking.

However, as per Tesler’s Law (or the Law of Conservation of Compexlity), this networking complexity is inherent - when you are finding real, runtime issues in larger scale clusters, it can often be at a very low networking level. These sorts of issues can be very hard to diagnose. I have started tracking some examples at the end of the article, but I think that Tinder’s Migration to Kuberenetes shows this challenge very well.

Overall - the transition is still likely to be for the best, but doesn’t come without some serious challenges at the networking level, which will require some serious expertise to deal with!

The Death of Microservice Madness!

Avoid the madness by making careful and considered decisions. To help out on this I’ve noted a few questions you might want to ask yourself, and what the answers might indicate:

You can download a PDF copy here: microservice-questions.pdf

Final Thoughts: Don’t Confuse Microservices with Architecture

I’ve deliberately avoided the ‘a’ word in this article. But my friend Zoltan made a very good point when proofing this article (which he has contributed to).

There is no microservice architecture. Microservices are just another pattern or implementation of components, nothing more, nothing less. Whether they are present in a system or not does not mean that the architecture of the system is solved.

Microservices relate in many ways more to the technical processes around packaging and operations rather than the intrinsic design of the system. Appropriate boundaries for components continues to be one of the most important challenges in engineering systems.

Regardless of the size of your services, whether they are in Docker containers or not, you will always need to think carefully about how to put a system together. There are no right answers, and there are a lot of options.

I hope you found this article interesting! As always, please do comment below if you have any questions or thoughts. You can also follow some lively discussions on:

Appendix: Further Reading

The following links might be of interest:

Martin Fowler - Bounded Context - Martin’s articles are great, I’d thoroughly recommend this.
Martin Fowler - Microservices - An often recommended introduction to the pattern.
Microservices - Good or Bad? - Björn Frantzén’s thoughts on microservices, after reading this article.
When Not To Do Microservices - Excellent post on the topic from Christian Posta
Sean Hull - 30 questions to ask a serverless fanboy - Interesting thoughts on the challenges of serverless, from a serverless fan!
Dave Kerr - Monoliths to Microservices - Practical tips for CI/CD and DevOps in the Microservice world - A recent conference presentation I did on devops with microservices.
Alexander Yermakov - Microservices without fundamentals - A response to this article, with Alex’s thoughts and counterpoints to the points raised here (see also Microservices as a self sufficient concept)

Please do share anything else you think makes great reading or watching on the topic!

Thanks

Thanks José from campusmvp.es for having the article translated in Spanish - La muerte de la locura de los microservicios en 2018!

Case Studies

Some interesting examples of experiences I am collecting of larger organisations who have made large scale transitions to microservices:

Tinder’s Move to Kubernetes

References

https://trends.google.com/trends/explore?date=today%205-y&q=microservice ↩︎
If you don’t want to miss the article, you can subscribe to the RSS Feed, or follow me on LinkedIn or Twitter. ↩︎
Docker Compose is a good solution, Fuge is very clever, and there is also the option of running orchestration locally as is the case with something like MiniKube. ↩︎
Google Kubernetes Engine, a managed service from Google Cloud Platform for Kubernetes: https://cloud.google.com/kubernetes-engine/ ↩︎
Amazon Elastic Container Services for Kubernetes, a managed service from Amazon Web Services for Kubernetes: https://aws.amazon.com/eks/ ↩︎

Simple Continuous Integration for Docker Images

Thu, 03 Nov 2016 05:14:35 +0000

In this article I’m going to demonstrate a few tips and tricks which can make your life easier when you are building or maintaining Dockerfiles.

The need for a Build Pipeline

Do we really need any kind of continuous integration or build pipeline for Dockerfiles?

There will be cases when the answer is no. However, if the answer to any of the following questions is ‘yes’, it might be worth considering:

Do you want others to be able to contribute to the Dockerfile, perhaps changing the image over time?
Are there specific functionalities in your Dockerfiles which could break if altered?
Do you expect to need to release updates to your Dockerfile?

Essentially, if we are looking at providing some kind of automated quality assurance and automation around building and releasing, then a build pipeline is not a bad idea.

A simple Build Pipeline

Here’s what a simple build pipeline could look like. This example is for a Docker Image I just created for local DynamoDB development - dwmkerr/docker-dynamodb:

Let’s dissect what we’ve got here.

The Dockerfile

This is the main ‘code’ of the project if you like. The Dockerfile is the recipe for the image we create.

The Continuous Integration Service

In this case, I am using CircleCI, however the approach described would work fine with most CI systems (such as Jenkins, TravisCI and TeamCity). There is an option to use the Docker Hub Automated Builds, but I’ve found this doesn’t give the flexibility I need (see Why not Docker Hub Automated Builds).

Essentially the CI service needs to offer the option to have three distinct steps in the pipeline, each of which must pass for process to proceed:

Build
Test
Deploy

The Build

We can build with tools, script files, whatever. At the moment, I am leaning towards makefiles. Normally I only need a few lines of shell script to do a build - anything more complex and the makefile can call a shell script. See also Why Makefiles?

Here’s what it might look like:

build:
docker build -t dwmkerr/dynamodb:latest .
ifndef BUILD_NUM
$(warning No build number is defined, skipping build number tag.)
else
docker build -t dwmkerr/dynamodb:$(BUILD_NUM) .
endif

This command just builds the Dockerfile and tags it as dwmkerr/dynamodb:lastest. If a BUILD_NUM variable is present, we also create the tag dwmkerr/dynamodb:BUILD_NUM. This means if we want to deploy to a service such as Amazon ECS we can push a specific build by referring to the image with that tag.

The Tests

Again I’m relying on make. I just want to be able to run make test - if zero is returned I’m happy. If not, the pipeline should stop and I’ll check the output. Here’s my test command:

test: build
./test/basics.test.sh
./test/ephemeral.test.sh
./test/persistent.test.sh

Not a thing of beauty, but it works. These scripts I’ll discuss a little bit later on, in the delightly titled What are these test scripts section.

For CircleCI, this is enough to have the main part of our pipeline. Here’s how the circle.yml file looks at this stage:

machine:
services:
- docker
environment:
# Set the build number, used in makefiles.
BUILD_NUM: $CIRCLE_BUILD_NUM
test:
override:
- make test

(Actually there’s a couple of other bits but they’re just to make sure circle uses the right version of Docker, see the full circle.yml file here).

The Deployments

Deployments are trivial as all we need to do is push to the Docker Hub. The make deploy command looks-a like this:

deploy:
docker push dwmkerr/dynamodb:latest
ifndef BUILD_NUM
$(warning No build number is defined, skipping push of build number tag.)
else
docker push dwmkerr/dynamodb:$(BUILD_NUM)
endif

We’re pushing the latest tag and BUILD_NUM tag if present. To add this to the CircleCI pipeline, we just add the following to circle.yml:

deployment:
master:
branch: master
commands:
- docker login -e $DOCKER_EMAIL -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
- make deploy

If we have a push to master, we log in to Docker (using environment variables I configure in the CircleCI UI) and then run make deploy to push our images.

That’s It

That’s about it. This is a pretty simple approach, you can see it in action at:

github.com/dwmkerr/docker-dynamodb

The rest of this post is a bit of a deep dive into some specific areas I found interesting.

Appendix 1: Why not Docker Hub Automated Builds?

There are automated builds available in the Docker Hub:

I’m not using this feauture at the moment, here’s a brief roundup of what I think are the current pros and cons:

Pros

You don’t have to goof around installing Docker on a CI platform.
It allows you to update the description of your Docker image automatically, from the GitHub README.md.
It allows you to associate the image with a specific GitHub repo (rather than just linking from the image description).
Branch management - allowing tags to be built for specific branches.

Cons

It doesn’t seem to support any kind of configurable gating, such as a running a test command prior to deploying.
It doesn’t seem to support any kind of triggering of downstream processes, such as updating environments, sending notifications or whatever.

The lack of ability to perform tests on the image before deploying it why I’m currently not using the service.

By doing the testing in a CI system for every pull request and only merging PRs where the tests pass we could mitigate the risk here. This service is worth watching as I’m sure it will evolve quickly.

Appendix 2: Why Makefiles?

I started coding with a commandline compiler in DOS. When I used my first GUI (Borland Turbo C++) it felt like a huge leap:

Later on I moved onto Microsoft Visual C++ 4.2:

And you cannot imagine the excitement when I got my boxed edition of Visual Studio .NET:

Wow!

Anyway, I digress. GNU make was invented by Leonardo Da Vinci in 1473 to allow you to build something from the commandline, using a fairly consistent syntax.

It is near ubiquitous on *nix systems. I am increasingly using it as an ’entry point’ to builds, as I use variety of languages and platforms. Being able to know that most of the time:

make build
make test

Will build and test something is convenient. Makefiles actually are not that great to work with (see this, this and this). I’ve found as long as you keep the commands simple, they’re OK. For anything really complex, I normally have a scripts/ folder, but call the scripts from the makefile, so that there’s still a simple entrypoint.

I’m not entirely sold on makefiles, but they tend to be my default at the moment if I know I’m going to use the commandline for builds (for example, in Java projects I’ll often write a makefile to call Maven or Gradle).

For things like Node.js, where you have commands like npm test or npm run xyz I still sometimes use makefiles, using npm for day-to-day dev tests (npm start) and make if it’s something more complex (e.g. make deploy-sit to deploy to an SIT environment).

Appendix 3: What are these test scripts?

You may have noticed:

test: build
./test/basics.test.sh
./test/ephemeral.test.sh
./test/persistent.test.sh

What’s going on here?

My Docker image is just a wrapper around Amazon’s Local DynamoDB tool. I don’t really need to test that tool. But what I wanted to test was the capabilities which lie at the intersection between ’native’ Docker and ’native’ DynamoDB.

For example, I know Docker supports volume mapping. I know DynamoDB supports using a data directory, to allow persistent between runs. I want to test I can combine Docker volume mapping and the DynamoDB data directory features. I know Docker images should default to being ephemeral, I want to test this holds true by default for my image.

Testing Docker is a little hard - I want to test that I can run containers, start, stop, check state before and after and so on. This is essentially an integration test, it can be tricky to make it truly isolated and deterministic.

I’ve given it my best go with these scripts. Here’s an example for the ’ephemeral’ test, where I’m trying to assert that if I run a container, create a table, stop the container and run a new one, I no longer have the table. Here’s the test:

# Bomb if anything fails.
set -e

# Kill any running dynamodb containers.
echo "Cleaning up old containers..."
docker ps -a | grep dwmkerr/dynamodb | awk '{print $1}' | xargs docker rm -f || true

# Run the container.
echo "Checking we can run the container..."
ID=$(docker run -d -p 8000:8000 dwmkerr/dynamodb)
sleep 2

# Create a table.
aws dynamodb --endpoint-url http://localhost:8000 --region us-east-1 \
 create-table \
 --table-name Supervillains \
 --attribute-definitions AttributeName=name,AttributeType=S \
 --key-schema AttributeName=name,KeyType=HASH \
 --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

# Clean up the container. On CircleCI the FS is BTRFS, so this might fail...
echo "Stopping and restarting..."
docker stop $ID && docker rm $ID || true
ID=$(docker run -d -p 8000:8000 dwmkerr/dynamodb)
sleep 2

# List the tables - there shouldn't be any!
COUNT=$(aws dynamodb --endpoint-url http://localhost:8000 --region us-east-1 \
 list-tables \
 | jq '.TableNames | length')
if [ $COUNT -ne "0" ]; then
 echo "Expected to find no tables, found $COUNT..."
 exit 1
fi

It’s a bit dirty - it removes containers from the host, changes things and so on. But it works.

I did experiment with running these tests in a container, which has the benefit of giving you a clean host to start with, which you can throw away after each test.

I had to give up after a little while due to time constraints, but will probably revisit this process. The benefits of running these integration tests in a container is that we get a degree of isolation from the host.

If anyone is interested, my attempts so far are on this RFC Pull Request. Feel free to jump in!

Learn Docker by building a Microservice

Tue, 19 Apr 2016 08:54:39 +0000

If you are looking to get your hands dirty and learn all about Docker, then look no further!

In this article I’m going to show you how Docker works, what all the fuss is about, and how Docker can help with a basic development task - building a microservice.

We’ll use a simple Node.js service with a MySQL backend as an example, going from code running locally to containers running a microservice and database.

Once you’ve read the article, you can find the source code here:

github.com/dwmkerr/node-docker-microservice

What is Docker?

At its heart, Docker is software which lets you create an image (which is a lot like a template for a virtual machine) and then run instances of that image in a container.

Docker maintains a vast repository of images, called the Docker Hub which you can use as starting points or as free storage for your own images. You can install Docker, choose an image you’d like to use, then run an instance of it in a container.

We’re going to build images, create containers from images and more in this article.

Install Docker

To follow along and use this article, you’ll need Docker.

Check the installation guide for your platform, docs.docker.com/engine/installation.

If you are on Mac or Windows, consider using a Virtual Machine. I use Parallels on Mac OS X to run an Ubuntu machine for most development activities. Being able to take snapshots, break things and then revert back is very handy when experimenting.

Try It Out

Enter this command:

docker run -it ubuntu

After a bit of spinning, you’ll see a prompt like this:

root@719059da250d:/#

Try out a few commands and then exit the container:

root@719059da250d:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
root@719059da250d:/# exit

This doesn’t look like much, but a lot has happened!

What you are seeing is the bash shell of an isolated container running Ubuntu, on your machine. It’s yours to place with - you can install things on it, run software, whatever you want.

Here’s a diagram and breakdown of what just happened (the digram is from the ‘Understanding the Architecture’ Docker Documentation, which is great):

We issue a docker command:

docker: run the docker client
run: the command to run a new container
-it: option to give the container an interactive terminal
ubuntu: the image to base the container on

The docker service running on the host (our machine) checks to see if we have a copy of the requested image locally- which there isn’t.
The docker service checks the public registry (the docker hub) to see if there’s an image named ubuntu available- which there is.
The docker service downloads the image and stores it in its local cache of images (ready for next time).
The docker service creates a new container, based on the ubuntu image.

Try any of these:

docker run -it haskell
docker run -it java
docker run -it python

We’re not going to use Haskell today, but you can see, running an environment is very easy.

It’s a snap to build images of our own, with our apps or services on them, databases, whatever we need. We can then run them on any machine with Docker installed - and the image will run in the same, predictable way. We can build our software and the environment it runs on as code and deploy easily.

Let’s look into a simple microservice as an example.

The Brief

We’re going to build a microservice which lets us manage a directory of email addresses to phone numbers, using Node.js and MySQL.

Getting Started

For doing local development we’ll need to install MySQL and create a test database for us to…

…nope.

Creating a local database and running scripts on it is an easy start, but can get messy. Lots of uncontrolled stuff going on. It might work, we could even control it with some shell scripts checked in to our repo, but what if other developers already have MySQL installed? What if they have a database already with the creative name ‘users’ which we want to create?

Step 1: Creating a Test Database Server - in Docker

This is a great Docker use case. We might not want to run our production database in Docker (perhaps we’ll just use Amazon RDS for example), but we can spin up a clean MySQL database in no time as a Docker container for development - leaving our development machine clean and keeping everything we do controlled and repeatable.

Run the following command:

docker run --name db -d -e MYSQL_ROOT_PASSWORD=123 -p 3306:3306 mysql:latest

This starts a MySQL instance running, allowing access through port 3306 using the root password 123.

docker run tells the engine we want to run an image (the image comes at the end, mysql:vlatest
--name db names this container db.
-d (or --detach) detach - i.e. run the container in the background.
-e MYSQL_ROOT_PASSWORD=123 (or --env) environment variables - tells docker we want to provide an environment variable. The variable following it is what the MySQL image checks for setting the default root password.
-p 3306:3306 (or --publish tells the engine that we want to map the port 3306 from inside the container to out port 3306.

The last part is really important - even though that’s the MySQL default port, if we don’t tell docker explicitly we want to map it, it will block access through that port (because containers are isolated until you tell them you want access).

The return value of this function is the container id, a reference to the container which you can use to stop it, restart it, issue commands on it and so on. Let’s see which containers are running:

$ docker ps
CONTAINER ID IMAGE ... NAMES
36e68b966fd0 mysql:latest ... db

The key information is the container ID, image and name. Let’s connect to this image and see what’s there:

$ docker exec -it db /bin/bash
root@36e68b966fd0:/# mysql -uroot -p123
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
+--------------------+
1 rows in set (0.01 sec)
mysql> exit
Bye
root@36e68b966fd0:/# exit

This is pretty clever too:

docker exec -it db tells docker we want to execute a command on the container named db (we could also use the id, or just the first few letters of the id). -it ensures we have an interactive terminal.
mysql -uroot -p123 the command we actually run as a process in the container, which in this case is just the mysql client.

We can create databases, tables, users, whatever we need.

Wrapping up the Test Database

Running MySQL inside a container has already introduced a few Docker tricks, but let’s pause here and move onto the service. For now, we’ll have create a test-database folder with a script to start the database, stop the database and setup test data:

test-database\setup.sql
test-database\start.sh
test-database\stop.sh

Start is simple:

#!/bin/sh

# Run the MySQL container, with a database named 'users' and credentials
# for a users-service user which can access it.
echo "Starting DB..."
docker run --name db -d \
 -e MYSQL_ROOT_PASSWORD=123 \
 -e MYSQL_DATABASE=users -e MYSQL_USER=users_service -e MYSQL_PASSWORD=123 \
 -p 3306:3306 \
 mysql:latest

# Wait for the database service to start up.
echo "Waiting for DB to start up..."
docker exec db mysqladmin --silent --wait=30 -uusers_service -p123 ping || exit 1

# Run the setup script.
echo "Setting up initial data..."
docker exec -i db mysql -uusers_service -p123 users < setup.sql

This script runs the database image in a detached container (i.e. in the background), with a user set up to access a users database, then waits for the database server to start up, then runs a setup.sql script to set initial data.

setup.sql is:


create table directory (user_id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, email TEXT, phone_number TEXT);
insert into directory (email, phone_number) values ('homer@thesimpsons.com', '+1 888 123 1111');
insert into directory (email, phone_number) values ('marge@thesimpsons.com', '+1 888 123 1112');
insert into directory (email, phone_number) values ('maggie@thesimpsons.com', '+1 888 123 1113');
insert into directory (email, phone_number) values ('lisa@thesimpsons.com', '+1 888 123 1114');
insert into directory (email, phone_number) values ('bart@thesimpsons.com', '+1 888 123 1115');

The stop.sh script will stop the container and remove it (containers are left around by docker by default so that they can be restared quickly, we don’t really need that feature for this example):

#!/bin/sh

# Stop the db and remove the container.
docker stop db && docker rm db

We’re going to make this even more slick later on, simplifying this further. Check the code at this stage by looking at the step1 branch of the repo.

Step 2: Creating a Microservice in Node.js

This article is really focused on learning Docker, so I’m not going to spend ages on the Node.js microservice. Instead, I’ll highlight the areas and takeaways.

test-database/ # contains the code seen in Step 1
users-service/ # root of our node.js microservice
- package.json # dependencies, metadata
- index.js # main entrypoint of the app
- api/ # our apis and api tests
- config/ # config for the app
- repository/ # abstraction over our db
- server/ # server setup code

Let’s take this apart bit by bit. The first section to look at is repository. It can be useful to wrap your database access in some kind of class or abstraction, to allow to mock it for testing purposes:

// repository.js
//
// Exposes a single function - 'connect', which returns
// a connected repository. Call 'disconnect' on this object when you're done.
'use strict';

var mysql = require('mysql');

// Class which holds an open connection to a repository
// and exposes some simple functions for accessing data.
class Repository {
 constructor(connection) {
 this.connection = connection;
 }

 getUsers() {
 return new Promise((resolve, reject) => {

 this.connection.query('SELECT email, phone_number FROM directory', (err, results) => {
 if(err) {
 return reject(new Error("An error occured getting the users: " + err));
 }

 resolve((results || []).map((user) => {
 return {
 email: user.email,
 phone_number: user.phone_number
 };
 }));
 });

 });
 }

 getUserByEmail(email) {

 return new Promise((resolve, reject) => {

 // Fetch the customer.
 this.connection.query('SELECT email, phone_number FROM directory WHERE email = ?', [email], (err, results) => {

 if(err) {
 return reject(new Error("An error occured getting the user: " + err));
 }

 if(results.length === 0) {
 resolve(undefined);
 } else {
 resolve({
 email: results[0].email,
 phone_number: results[0].phone_number
 });
 }

 });

 });
 }

 disconnect() {
 this.connection.end();
 }
}

// One and only exported function, returns a connected repo.
module.exports.connect = (connectionSettings) => {
 return new Promise((resolve, reject) => {
 if(!connectionSettings.host) throw new Error("A host must be specified.");
 if(!connectionSettings.user) throw new Error("A user must be specified.");
 if(!connectionSettings.password) throw new Error("A password must be specified.");
 if(!connectionSettings.port) throw new Error("A port must be specified.");

 resolve(new Repository(mysql.createConnection(connectionSettings)));
 });
};

There’s probably a lot of better ways to do this! But basically we can create a Repository object like this:

repository.connect({
 host: "127.0.0.1",
 database: "users",
 user: "users_service",
 password: "123",
 port: 3306
}).then((repo) => {
 repo.getUsers().then(users) => {
 console.log(users);
 });
 repo.getUserByEmail('homer@thesimpsons.com').then((user) => {
 console.log(user);
 })
 // ...when you are done...
 repo.disconnect();
});

There’s also a set of unit tests in the repository/repository.spec.js file. Now that we’ve got a repo, we can create a server. This is server/server.js:

// server.js

var express = require('express');
var morgan = require('morgan');

module.exports.start = (options) => {

 return new Promise((resolve, reject) => {

 // Make sure we have a repository and port provided.
 if(!options.repository) throw new Error("A server must be started with a connected repository.");
 if(!options.port) throw new Error("A server must be started with a port.");

 // Create the app, add some logging.
 var app = express();
 app.use(morgan('dev'));

 // Add the APIs to the app.
 require('../api/users')(app, options);

 // Start the app, creating a running server which we return.
 var server = app.listen(options.port, () => {
 resolve(server);
 });

 });
};

This module exposes a start function, which we can use like this:

var server = require('./server/server);
server.start({port: 8080, repo: repository}).then((svr) => {
 // we've got a running http server :)
});

Notice that server.js uses api/users/js? Here it is:

// users.js
//
// Defines the users api. Add to a server by calling:
// require('./users')
'use strict';

// Only export - adds the API to the app with the given options.
module.exports = (app, options) => {

 app.get('/users', (req, res, next) => {
 options.repository.getUsers().then((users) => {
 res.status(200).send(users.map((user) => { return {
 email: user.email,
 phoneNumber: user.phone_number
 };
 }));
 })
 .catch(next);
 });

 app.get('/search', (req, res) => {

 // Get the email.
 var email = req.query.email;
 if (!email) {
 throw new Error("When searching for a user, the email must be specified, e.g: '/search?email=homer@thesimpsons.com'.");
 }

 // Get the user from the repo.
 options.repository.getUserByEmail(email).then((user) => {

 if(!user) {
 res.status(404).send('User not found.');
 } else {
 res.status(200).send({
 email: user.email,
 phoneNumber: user.phone_number
 });
 }
 })
 .catch(next);

 });
};

Both of these files have unit tests adjacent to the source.

We’ll need config. Rather than using a specialised library, a simple file will do the trick - config/config.js:

// config.js
//
// Simple application configuration. Extend as needed.
module.exports = {
 port: process.env.PORT || 8123,
 db: {
 host: process.env.DATABASE_HOST || '127.0.0.1',
 database: 'users',
 user: 'users_service',
 password: '123',
 port: 3306
 }
};

We can require config as needed. Currently, most config is hard coded, but as you can see from port it’s easy to add environment variables as an option.

Final step - stringing it together with an index.js file which composes everything:

// index.js
//
// Entrypoint to the application. Opens a repository to the MySQL
// server and starts the server.
var server = require('./server/server');
var repository = require('./repository/repository');
var config = require('./config/config');

// Lots of verbose logging when we're starting up...
console.log("--- Customer Service---");
console.log("Connecting to customer repository...");

// Log unhandled exceptions.
process.on('uncaughtException', function(err) {
 console.error('Unhandled Exception', err);
});
process.on('unhandledRejection', function(err, promise){
 console.error('Unhandled Rejection', err);
});

repository.connect({
 host: config.db.host,
 database: config.db.database,
 user: config.db.user,
 password: config.db.password,
 port: config.db.port
}).then((repo) => {
 console.log("Connected. Starting server...");

 return server.start({
 port: config.port,
 repository: repo
 });

}).then((app) => {
 console.log("Server started successfully, running on port " + config.port + ".");
 app.on('close', () => {
 repository.disconnect();
 });
});

We have a little error handling and beyond that we’re just loading config, creating a repo and starting our server.

That’s the microservice. It allows us to get all users, or search a user:

HTTP GET /users # gets all users
HTTP GET /search?email=homer@thesimpons.com # searches by email

If you checkout the code, you’ll see that there’s a few commands available for you:

cd ./users-service
npm install # setup everything
npm test # unit test - no need for a test database running
npm start # run the server - you must have a test database running
npm run debug # run the server in debug mode, opens a browser with the inspector
npm run lint # check to see if the code is beautiful

Asides from the code you’ve seen we have:

Node Inspector for debugging
Mocha/shoud/supertest for unit tests
ESLint for linting

That’s it!

Run the test database with:

cd test-database/
./start.sh

Then the service with:

cd ../users-service/
npm start

You can point your browser to localhost:8123/users and see it in action. If you are using Docker Machine (i.e. you’re on Mac or Windows) then localhost won’t work, you need the IP of the docker machine instead. You can use docker-machine ip to get it.

We’ve whipped through building the service quickly. If you’d like to see this code before we continue, check the step2 branch.

Step 3: Dockerising our Microservice

OK now it gets fun!

So we have a microservice which we can run on a dev box, as long as it has a compatible version of Node.js installed. What we’d like to do is set up our service so that we can create a Docker Image from it, allowing us to deploy our service anywhere which supports docker.

The way we do this is create a Dockerfile. A Dockerfile is a recipe that tells the Docker engine how to build your image. We’ll create a simple Dockerfile in our users-service directory and start to explore how we can adapt it to our needs.

Creating the Dockerfile

Create a new text file called Dockerfile at users-service/ with the content below:

# Use Node v4 as the base image.
FROM node:4
# Run node
CMD ["node"]

Now run the commands below to build the image and run the a container from it:

docker build -t node4 . # Builds a new image
docker run -it node4 # Run a container with this image, interactive

Let’s look at the build command first.

docker build tell the engine we want to create a new image.
-t node4 tag this image with the tag node4. We can refer to this image by tag from now on.
. use the current directory to find the Dockerfile.

After some console output you’ll see we have a new image created. You can see all images on your system with docker images. The next command should be fairly familiar from what we’ve done so far:

docker run run a new container from an image.
-it use an interactive terminal.
node4 the tag of the image we want to use in the container.

When we run this image, we get a node repl, check the current version like so:

> process.version
'v4.4.0'
> process.exit(0)

This is potentially different to the node version on your current machine.

Examining the Dockerfile

Looking at the dockerfile we can see quite easily what is going on:

FROM node:4 the first thing we specify in a Dockerfile is the base image. A quick google finds the node organisation page on the docker hub showing all of the available images. This is essentially bare bones ubuntu with node installed.
CMD ["node"] the CMD command tells docker that this image should run the node executable. When the executable terminates, the container shuts down.

With the addition of a few more commands, we can update our Dockerfile so that it runs our service:

# Use Node v4 as the base image.
FROM node:4
# Add everything in the current directory to our image, in the 'app' folder.
ADD . /app
# Install dependencies
RUN cd /app; \
npm install --production
# Expose our server port.
EXPOSE 8123
# Run our app.
CMD ["node", "/app/index.js"]

The only addition here is that we use the ADD command to copy everything¹ in the current directory to a folder in the container called app/ . We then use RUN to run a command in the image, which installs our modules. Finally, we EXPOSE the server port, telling docker we intend to support inbound connections on 8123, then run our server code.

Ensure the test-database service is running, then build and run the image again:

docker build -t users-service .
docker run -it -p 8123:8123 users-service

If you navigate to localhost:8123/users in a browser you should see an error, checking the console shows our container is reporting some issues:

--- Customer Service---
Connecting to customer repository...
Connected. Starting server...
Server started successfully, running on port 8123.
GET /users 500 23.958 ms - 582
Error: An error occured getting the users: Error: connect ECONNREFUSED 127.0.0.1:3306
at Query._callback (/app/repository/repository.js:21:25)
at Query.Sequence.end (/app/node_modules/mysql/lib/protocol/sequences/Sequence.js:96:24)
at /app/node_modules/mysql/lib/protocol/Protocol.js:399:18
at Array.forEach (native)
at /app/node_modules/mysql/lib/protocol/Protocol.js:398:13
at nextTickCallbackWith0Args (node.js:420:9)
at process._tickCallback (node.js:349:13)

Yikes! So the connection from our users-service container to the test-database container is being refused. We might try running docker ps to see all containers running:

CONTAINER ID IMAGE PORTS NAMES
a97958850c66 users-service 0.0.0.0:8123->8123/tcp kickass_perlman
47f91343db01 mysql:latest 0.0.0.0:3306->3306/tcp db

They’re both there, so what is going on?

Linking Containers

The issue we’ve seen is actually to be expected. Docker containers are supposed to be isolated, so it wouldn’t make much sense if we could create connections between containers without us explicitly allowing it.

Yes, we can connect from our machine (the host) to a container, because we’ve opened ports for that (using the -p 8123:8123 argument for example). If we allowed containers to talk to each other in the same way, then two containers running on the same machine would be able to communicate, even if the developers didn’t intend it, and that’s a recipe for disaster, especially when we might have a cluster of machines whos job it is to run containers from different applications.

If we’re going to connect from one container to another, we need to link them, which tells docker that we explicitly want to allow communication between the two. There are two ways of doing this, the first is the ‘old fasioned’ but quite simple way, the second we’ll see a little later.

Linking Containers with the ’link’ parameter

When we run a container, we can tell docker that we intend to connect to another container using the link parameter. In our case, we can run our service correctly like this:

docker run -it -p 8123:8123 --link db:db -e DATABASE_HOST=DB users-service

docker run -it run a docker image in a container, with an interactive terminal.
-p 8123:8123 map the host port 8123 to the container port 8123.
link db:db link to the container named db and refer to it as db.
-e DATABASE_HOST=db set the DATABASE_HOST environment variable to db.
users-service the name of the image to run in our container.

Now when we go to localhost:8123/users everything works.

How it works

Remember our config file for the service? It allowed us to specify a database host with an environment variable:

// config.js
//
// Simple application configuration. Extend as needed.
module.exports = {
 port: process.env.PORT || 8123,
 db: {
 host: process.env.DATABASE_HOST || '127.0.0.1',
 database: 'users',
 user: 'users_service',
 password: '123',
 port: 3306
 }
};

When we run the container, we set this environment variable to DB, which means we’re connecting to a host called DB. This is automatically set up for us by the docker engine when we link to a container.

To see this in action, try running docker ps to list all running containers. Look up the name of the container running the users-service, which will be a random name such as trusting_jang:

docker ps
CONTAINER ID IMAGE ... NAMES
ac9449d3d552 users-service ... trusting_jang
47f91343db01 mysql:latest ... db

Now we can look at the hosts available on our container:

docker exec trusting_jang cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2 db 47f91343db01 # linking magic!!
172.17.0.3 ac9449d3d552

Remember how docker exec works? Choose a container name and then whatever follows is the command you’ll execute on the container, in our case cat /etc/hosts.

OK the hosts file doesn’t have the # linking magic!! comment, that’s so you can see - docker has added db to our hosts file so we can refer to the linked container by hostname. This is one consequence of linking. Here’s the other:

docker exec trusting_jang printenv | grep DB
DB_PORT=tcp://172.17.0.2:3306
DB_PORT_3306_TCP=tcp://172.17.0.2:3306
DB_PORT_3306_TCP_ADDR=172.17.0.2
DB_PORT_3306_TCP_PORT=3306
DB_PORT_3306_TCP_PROTO=tcp
DB_NAME=/trusting_jang/db

From this command we can also see that when docker links a container, it also provides a set of environment variables with some helpful information. We know the host, tcp port and container name.

That’s step 3 complete - we have a MySQL database running happily in a container, we have a node.js microservice which we can run locally or in a container of its own, and we know how to link them together.

You can check out how the code looks at this stage by going to the step3 branch.

Step 4: Integration Testing the Environment

We can now write an integration test which calls the actual server, running as a docker container, calling the containerised test database.

Writing the integration test can be done in whatever language or on whatever platform you want, within reason, but to keep things simple I’m using Node.js as we’ve already seen Mocha and Supertest in our project.

In a new folder, called integration-tests we’ve got a single index.js:

var supertest = require('supertest');
var should = require('should');

describe('users-service', () => {

 var api = supertest('http://localhost:8123');

 it('returns a 200 for a known user', (done) => {

 api.get('/search?email=homer@thesimpsons.com')
 .expect(200, done);
 });

});

This will check an API call and show the results of the test².

As long as your users-service and test-database are running, the tests will pass. However, at this stage the services are getting a little harder to handle:

We have to use a shell script to start and stop the database
We have to remember a sequence of commands to start the users service against the database
We have to use node directly to run the integration tests

Now that we’re a little more familiar with Docker we can fix these issues.

Simplifiying the Test Database

Currently we have the following files for the test database:

/test-database/start.sh
/test-database/stop.sh
/test-database/setup.sql

Now that we’re more familar with Docker, we can improve on this. Looking into the mysql image documentation on Docker Hub there’s a note which tells us any .sql or .sh file added to the image’s /docker-entrypoint-initdb.d folder will be executed when setting up the DB.

This means we can replace our start.sh and stop.sh scripts with a Dockerfile:

FROM mysql:5
ENV MYSQL_ROOT_PASSWORD 123
ENV MYSQL_DATABASE users
ENV MYSQL_USER users_service
ENV MYSQL_PASSWORD 123
ADD setup.sql /docker-entrypoint-initdb.d

Now to run our test database it is just:

docker build -t test-database .
docker run --name db test-database

Composing

Building and running each container is still somewhat time consuming. We can take things a step further with the Docker Compose tool.

Docker Compose lets you create a file which defines each container in your system, the relationships between them, and build or run them all.

First, install Docker Compose. Now create a new file in the root of your project called docker-compose.yml:

version: '2'
services:
users-service:
build: ./users-service
ports:
- "8123:8123"
depends_on:
- db
environment:
- DATABASE_HOST=db
db:
build: ./test-database

Now check this out:

docker-compose build
docker-compose up

Docker Compose has built all of the images needed for our application, created containers fromthem, run them in the correct order and started the whole stack!

The docker-compose build command builds each image which is listed in the docker-compose.yml file:

version: '2'
services:
users-service:
build: ./users-service
ports:
- "8123:8123"
depends_on:
- db
environment:
- DATABASE_HOST=db
db:
build: ./test-database

The build value for each of our services tells docker where to go to find the Dockerfile. When we run docker-compose up, docker starts all of our services. Notice from the Dockerfile we can specify ports and dependencies. Actually, there’s a whole bunch of config we can change here.

In another terminal, run docker compose down to gracefully shut down the containers.

Winding Up

We’ve seen a lot of docker in this article, but there’s a lot more to it. I hope this has shown some of the interesting and useful things that you can use docker for in your workflow.

As usual, questions and comments are welcomed! I’d also strongly recommend the document Understanding Docker to get a deeper understanding of how docker works.

You can see the final source code for the project built in this article at github.com/dwmkerr/node-docker-microservice

Notes

Copying everything is actually a bad idea, because we will also copy the node_modules folder. Generally it is a better idea explicitly list the files or folders you want to copy, or use a .dockerignore file, which works just like the .gitignore file. ↩︎
If the server isn’t running, it will actually show a rather annoying exception, due to a bug in supertest, see github.com/visionmedia/supertest/issues/314. ↩︎