Docker on dwmkerr.com

Observations, tips and tricks for the CKA certification

Thu, 04 Jun 2020 00:00:00 +0000

In this article I’ll share some observations, tips and tricks for the Linux Foundation’s “Certified Kubernetes Administrator certification and exam.

I’ve been operating Kubernetes in multiple environments for a few years now. I thought this would be an easy certification to get, but I was surprised by how hard it was!

I took this exam without doing any formal training, I mostly focused on the areas of the curriculum which I knew I was a little weak at. The task-based structure for the exam I thought was really excellent. It took me two attempts to pass, and I learnt a few things along the way.

Here I’ll share some thoughts on the certification which hopefully will be useful if you are considering taking it!

Tip: Do the right Certification!
Tip: Understand the Format!
Tip: Know your Vim
You need to know the architecture of Kubernetes
Tip: You Need to know Linux Sysadmin
Tip: “Dry Run” is your friend
Tip: Know how to troubleshoot networking
Tip: Nail the easy questions quickly
That’s it!

Tip: Do the right Certification!

The CKA exam tests administration and operation skills and techniques for Kubernetes. If you have set up and administered clusters before, this will likely not be too challenging. But if you’ve never set up a cluster by hand, troubleshot weird issues, fixed clusters and so on, then this is likely going to be very hard.

There is a certification which is much more geared towards developers who use Kubernetes, but don’t necessarily administer it - that’s the CKAD exam and might be the one to take if you are not too familiar with system administration.

Tip: Understand the Format!

This is not a multiple choice question exam. It’s a task based exam, meaning you have about 22 or so specific tasks to complete, in a web browser which has a terminal connected to a cluster.

It is open-book - meaning that you can use the Kubernetes Documentation during the exam. It’s not a memory test of specific flags for commands or whatever, it will really require you to work with a running cluster. This means you’ll have to be pretty familiar with kubectl, kubeadm and also Linux in general!

Tip: Know your Vim

In the two exams I took, nano was available. But if you are using nano to work with files you may struggle for time.

I spent a lot of time in vim in the exam. vim is my main text editor for day to day work, so I’m fairly familiar with it. Knowing how to quickly copy a file (lets say for example a file which represents a deployment) and quickly manipulate the text in it will be crucial. Make sure you are going to be using a text editor which you can be efficient in!

You won’t be using a graphical text editor to work with files, so being competent in a terminal editor like vim or emacs could make a big difference. Of course you could install your favourite text editor, but you won’t be able to use a graphical editor like VS Code.

Also, as in most Linux distributions, screen is available out of the box, and tmux can also be installed. If you are familiar with either of these terminal mutliplexers it could save you a tonne of time, for example being able to run watch -n 5 -d kubectl get pods in one pane while applying resources in another.

You need to know the architecture of Kubernetes

This exam will require you to deal with trivial tasks such as running a deployment or creating a volume. But the questions which focus on that tend to only count for one or two percent of the overall grade each. Questions which deal with troubleshooting actual Kubernetes issues could count for six or seven percent each.

This means you need to know how Kubernetes is architecture. The kubelet which runs on nodes, the API server, the etcd store, all of these things you have to understand how they work and how they fit together.

The online documentation covers the architecture in detail, here’s the best place to start:

https://kubernetes.io/docs/concepts/overview/components/

You will need to know how the control plane works, how nodes communicate, how transport of messages works and is secured if you are going to have a chance at dealing with the harder questions.

Tip: You Need to know Linux Sysadmin

If you are not familiar with systemctl, journalctl, apt, systemd units and how the core Kubernetes components are configured, you’ll really struggle.

Look over the CNCF curriculum - expect to not just have to know how to deal with ‘happy path’ situations, but also broken clusters, incorrect configuration and so on.

Tip: “Dry Run” is your friend

One thing which helped me a lot in my second attempt at the exam was the --dry-run flag. Before you create resources or change anything, run the operation with the --dry-run flag and see whether the output is what you would expect.

This is a quick and easy way to see the changes to the cluster which you are going to apply - and troubleshoot them - before making any actual changes.

Tip: Know how to troubleshoot networking

Networking in Kubernetes is complex. You must be able to troubleshoot networking issues in the cluster to be able to deal with the more complex tasks.

This means that you should know how to be able to run typical networking tools like dig, nslookup, telnet etc, in the cluster itself.

If you are not familiar with these tools you might need to take an online course in Kubernetes or Linux Networking Administration before considering this certification. The Linux Certified Systems Administrator training would be a good place to start.

If you have taken the Docker Certified Associate exam then some of this should be familiar. If you are not very familiar with how Docker itself works, you’ll likely struggle with Kubernetes.

Tip: Nail the easy questions quickly

There are a lot of tasks which only count for one or two percent each; these ones you should be able to complete in a few minutes. You’ll need all the time in the exam to work on the really hard questions which deal with diagnosing and fixing cluster issues.

Know your core Kubernetes concepts; if you have done the CKAD exam you should be good, if not, check the curriculum and make sure you can quickly complete all of the trivial tasks without wasting too much time.

That’s it!

Hopefully this was helpful! Good luck if you are taking the exam and hopefully you’ll find it a challenging but rewarding experience. I’ve taken many exams over the years but this was one of the most challenging, but also one of the most enjoyable, I really felt like it was testing practical techniques rather than your ability to just remember random commands and flags.

As always, if you have any comments or questions, please just add them in the section below!

Manipulating Istio and other Custom Kubernetes Resources in Golang

Mon, 08 Oct 2018 21:34:02 +0000

In this article I’ll demonstrate how to use Golang to manipulate Kubernetes Custom Resources, with Istio as an example. No knowledge of Istio is needed, I’ll just use it to demonstrate the concepts!

Istio is a highly popular Service Mesh platform which allows engineers to quickly add telemetry, advanced traffic management and more to their service-based applications.

One interesting element of how Istio works is that when deployed into a Kubernetes cluster, many key configuration objects are handled as Custom Resources. Custom Resources are a very powerful Kubernetes feature, which allow you to create your own ‘first class’ resources (just like Pods, ReplicaSets, Deployments or whatever) and then interface with them using kubectl or the Kubernetes APIs.

In this article I’ll show you how to interface with these Custom Resources using the Golang Kubernetes client.

CRDs: A Quick Overview

When you set up Istio for your cluster, one common thing you will likely do is specify how you will route traffic. This can be quite sophisticated, as shown below:

Figure 1: Istio Traffic Management Examples, from istio.io

One way for a system like this to be configured would be to have a ConfigMap which contains the definition of how services are routed.

However, Istio actually registers new types of resources (Custom Resource Definitions) which represent things like Gateways or Services. We can create/update/delete/manipulate them just like any other Kubernetes object.

For example, I could create a virtual service for the example above with something like this:

cat << EOF | kubectl create -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: service2
spec:
 hosts:
 - "*"
 gateways:
 - demo1-gateway
 http:
 - route:
 - destination:
 host: service2
 subset: v1
 weight: 95
 - destination:
 host: service2
 subset: v2
 weight: 5
EOF

Again, the important thing is not the specific content of this resource, more the fact that I can treat my Istio resources just like I would any other Kubernetes object:

$ kubectl get virtualservices.networking.istio.io
NAME AGE
service2 93s

Or:

$ kubectl delete virtualservices.networking.istio.io/service2

I can use edit, describe, register lifecycle events, watch for changes, and so on.

Working with CRDs in Golang

The Golang Kubernetes Client allows you to create strongly defined types which you can then use to interface with CRDs. An example is in the Red Hat blog post Kubernetes Deep Dive: Code Generation for Custom Resources.

This is an excellent approach, but can feel pretty heavy if you want to quickly access some data, and don’t want to have to generate a lot of code.

There is an alternative, which is to use the DynamicClient. The preferred approach seems to be the first, which involves code generation, so little documentation exists for the second approach. However, it is actually very simple.

Here’s an example of how you can list all Istio VirtualService resources, without having to generate any code:

import (
 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/dynamic"
)

// Create a Dynamic Client to interface with CRDs.
dynamicClient, _ := dynamic.NewForConfig(config)

// Create a GVR which represents an Istio Virtual Service.
virtualServiceGVR := schema.GroupVersionResource{
 Group: "networking.istio.io",
 Version: "v1alpha3",
 Resource: "virtualservices",
}

// List all of the Virtual Services.
virtualServices, _ := dynamicClient.Resource(virtualServiceGVR).Namespace("default").List(metav1.ListOptions{})
for _, virtualService := range virtualServices.Items {
 fmt.Printf("VirtualService: %s\n", virtualService.GetName())
}

This snippet omits setup and error-handling for clarity, the full example is in the k8s-list-virtualservices.go gist.

Patching CRDs in Golang

You may have noticed that the .Resource().Namespace().List() code looks very similar to the structure for making API calls when using the Kubernetes Clientset. In fact, it is essentially the same. Looking at the interface, you can see you have all of the operations you’d expect:

Create
Update
Delete
Get

And so on. This is nice because you can use the same trick in my article ‘Patching Kubernetes Resources in Golang’ to manipulate these entities, without ever having to create a structure to represent it.

Here’s another abbreviated example, this time showing how we can adjust the weight of the routing from the services to 50%/50%:

import (
 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/dynamic"
)

// Create a GVR which represents an Istio Virtual Service.
virtualServiceGVR := schema.GroupVersionResource{
 Group: "networking.istio.io",
 Version: "v1alpha3",
 Resource: "virtualservices",
}

// Weight the two routes - 50/50.
patchPayload := make([]PatchUInt32Value, 2)
patchPayload[0].Op = "replace"
patchPayload[0].Path = "/spec/http/0/route/0/weight"
patchPayload[0].Value = 50
patchPayload[1].Op = "replace"
patchPayload[1].Path = "/spec/http/0/route/1/weight"
patchPayload[1].Value = 50
patchBytes, _ := json.Marshal(patchPayload)

// Apply the patch to the 'service2' service.
_, err := dynamicClient.Resource(virtualServiceGVR).Namespace("default").Patch("service2", types.JSONPatchType, patchBytes)

See the full example in the gist k8s-patch-virtualservice.go

After running the sample, you can use the Kubernetes CLI to verify the changes:

$ kubectl get virtualservices.networking.istio.io/service2 -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
clusterName: ""
creationTimestamp: 2018-10-08T09:53:16Z
generation: 0
name: service2
namespace: default
resourceVersion: "487435"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/service2
uid: fac5930c-cadf-11e8-90a2-42010a94005b
spec:
gateways:
- demo1-gateway
hosts:
- '*'
http:
- route:
- destination:
host: service2
subset: v1
weight: 50
- destination:
host: service2
subset: v2
weight: 50

Keep It Simple!

That’s it! This trick made something I was working on a lot easier, but it took a little bit of experimentation to get right. I hope you find the approach useful. Please share any thoughts/questions in the comments.

Patching Kubernetes Resources in Golang

Tue, 24 Jul 2018 06:33:17 +0000

Recently I needed to be able to quickly adjust the number of replicas in a Kubernetes Replication Controller. The original solution I’d seen pulled down the spec, modified it, then updated it. There’s a better way!

There’s a patch API for Kubernetes resources. Patching resources is faster and easier than pulling them and updating the spec wholesale. However, the documentation is a little limited.

After some trial and error I got it working, here’s the solution. I thought it might be helpful to share for others!

The Solution

I’ll start with the solution. If this is all you need, you are good to go. The details of how this works are presented afterwards. In this example I’ll update the number of replicas in the my-rc controller:

package main

import (
 "encoding/json"
 "fmt"

 types "k8s.io/apimachinery/pkg/types"
 "k8s.io/client-go/kubernetes"
 _ "k8s.io/client-go/plugin/pkg/client/auth"
 "k8s.io/client-go/tools/clientcmd"
)

var (
 // Leave blank for the default context in your kube config.
 context = ""

 // Name of the replication controller to scale, and the desired number of replicas.
 replicationControllerName = "my-rc"
 replicas = uint32(3)
)

// patchStringValue specifies a patch operation for a string.
type patchStringValue struct {
 Op string `json:"op"`
 Path string `json:"path"`
 Value string `json:"value"`
}

// patchStringValue specifies a patch operation for a uint32.
type patchUInt32Value struct {
 Op string `json:"op"`
 Path string `json:"path"`
 Value uint32 `json:"value"`
}

func scaleReplicationController(clientSet *kubernetes.Clientset, replicasetName string, scale uint32) error {
 payload := []patchUInt32Value{{
 Op: "replace",
 Path: "/spec/replicas",
 Value: scale,
 }}
 payloadBytes, _ := json.Marshal(payload)
 _, err := clientSet.
 CoreV1().
 ReplicationControllers("default").
 Patch(replicasetName, types.JSONPatchType, payloadBytes)
 return err
}

func main() {
 // Get the local kube config.
 fmt.Printf("Connecting to Kubernetes Context %v\n", context)
 config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
 clientcmd.NewDefaultClientConfigLoadingRules(),
 &clientcmd.ConfigOverrides{CurrentContext: context}).ClientConfig()
 if err != nil {
 panic(err.Error())
 }

 // Creates the clientset
 clientset, err := kubernetes.NewForConfig(config)
 if err != nil {
 panic(err.Error())
 }

 // Scale our replication controller.
 fmt.Printf("Scaling replication controller %v to %v\n", replicationControllerName, replicas)
 err = scaleReplicationController(clientset, replicationControllerName, replicas)
 if err != nil {
 panic(err.Error())
 }
}

This code is also available in the k8s-patch.go gist.

The Mechanism

The Kubernetes Patch API supports a few different methods for modifying resources. It is important to be aware that there is not a universally accepted ‘standard’ approach to representing a change to a resource in a REST API.

There are three strategies you can use to patch:

merge: follows the JSON Merge Patch Spec (RFC 7386)
stragetic: A strategic merge, which addresses some limitations of the merge patch (noted in this doc.
json: follows the JSON Patch Spec (RFC 6902)

These are documented in detail at:

docs/devel/api-conventions.md#patch-operations

The mechanism I’ve used here is json, which I think is the clearest to the reader. To use this strategy we need to build a payload describing what we are changing. This might look like this:

{
 "op": "replace",
 "path": "/spec/replicas",
 "value": 4
}

The op field can be remove, replace, add etc etc (all the details are in the RFC 6902), or the slightly more readable jsonpatch.com). This allows the operation to be very explicit to the reader, which is helpful. We create a struct which represents an operation on a string or integer (or whatever data type we need), serialize it and pass to the API.

Under the hood, the Golang client will simply translate this into an HTTP call which will look like something like this:

PATCH /api/v1/namespaces/default/replicationcontrollers/app-server-blue HTTP/1.1
Host: 127.0.0.1
Content-Type: application/json-patch+json
Content-Length: 70
[{
"op": "replace",
"path": "/spec/replicas",
"value": 4
}]

This corresponds to the documentation on the Patch Operations. Note that the patch operation type is specified in the Content-Type header.

Hopefully this’ll help you if you need to patch resources, are struggling with the docs and are a Go noob like me! Any tips on how to make the code cleaner or more idomatic would be welcome.

Thanks to the following articles and issues which helped me unpick this:

The Death of Microservice Madness in 2018

Fri, 12 Jan 2018 10:52:25 +0000

En Español | Reddit Thread | Hacker News Thread

Microservices became a very popular topic over the last couple of years¹. ‘Microservice madness’ goes something like this:

Netflix are great at devops. Netflix do microservices. Therefore: If I do microservices, I am great at devops.

There are many cases where great efforts have been made to adopt microservice patterns without necessarily understanding how the costs and benefits will apply to the specifics of the problem at hand.

I’m going to describe in detail what microservices are, why the pattern is so appealing, and also some of the key challenges that they present.

I’ll finish with a set of simple questions might be valuable to ask yourself when you are considering whether microservices are the right pattern for you. The questions are at the end of the article.

What are microservices, and why are they so popular?

Let’s start with the basics. Here is how a hypothetical video sharing platform might be implemented, first in the form of a monolith (single large unit) and then in the form of microservices:

The difference between the two systems is that the first is a single large unit; a monolith. The second is a set of small, specific services. Each service has a specific role.

When the diagram is drawn at this level of detail, it is easy to see the appeal. There are a whole host of potential benefits:

Independent Development: Small, independent components can be built by small, independent teams. A group can work on a change to the ‘Upload’ service without interfering with the ‘Transcode’ service, or even knowing about it. The amount of time to learn about a component is greatly reduced, and it is easier to develop new features.

Independent Deployment: Each individual component can be deployed independently. This allows new features to be released with greater velocity and less risk. Fixes or features for the ‘Streaming’ component can be deployed without requiring other components to be deployed.

Independent Scalability: Each component can be scaled independently of each other. During busy periods when new shows are released, the ‘Download’ component can be scaled up to handle the increased load, without having to scale up every component, which makes elastic scaling more feasible and reduces costs.

Reusability: Components fulfil a small, specific function. This means that they can more easily be adapted for use in other systems, services or products. The ‘Transcode’ component could be used by other business units, or even turned into a new business, perhaps offering transcoding services for other groups.

At this level of detail, the benefits of a microservice model over a monolithic model seem obvious. So if that’s the case - why is this pattern only recently in vogue? Where has it been all my life?

If this is so great, why hasn’t it been done before?

There are two answers to this question. One is that it has - to the best of our technical capabilities, and the other is that more recent technical advances have allowed us to take it to a new level.

When I started writing the answer to this question, it turned into a long description, so I’m actually going to separate it into another article and publish it a little later². At this stage, I will skip the journey from single program to many programs, ignore ESBs and Service Orientated Architecture, component design and bounded contexts, and so on.

Those who are interested can read more about the journey separately. Instead I’ll say that in many ways we’ve been doing this for a while, but with the recent explosion in popularity of container technology (Docker in particular) and in orchestration technology (such as Kubernetes, Mesos, Consul and so on) this pattern has become much more viable to implement from a technical standpoint.

So if we take it as a given that we can implement a microservice arrangement, we need to think carefully about the should. We’ve seen the high-level theoretical benefits, but what about the challenges?

What’s the problem with microservices?

If microservices are so great, what’s the big deal? Here are some of the biggest issues I’ve seen.

Increased complexity for developers

Things can get a lot harder for developers. In the case where a developer wants to work on a journey, or feature which might span many services, that developer has to run them all on their machine, or connect to them. This is often more complex than simply running a single program.

This challenge can be partially mitigated with tooling³, but as the number of services which makes up a system increases, the more challenges developers will face when running the system as a whole.

Increased complexity for operators

For teams who don’t develop services, but maintain them, there is an explosion in potential complexity. Instead of perhaps managing a few running services, they are managing dozens, hundreds or thousands of running services. There are more services, more communication paths, and more areas of potential failure.

Increased complexity for devops

Reading the two points above, it may grate that operations and development are treated separately, especially given the popularity of devops as a practice (which I am a big proponent of). Doesn’t devops mitigate this?

The challenge is that many organisations still run with separated development and operations teams - and a organisation that does is much more likely to struggle with adoption of microservices.

For organisations which have adopted devops, it’s still hard. Being both a developer and an operator is already tough (but critical to build good software), but having to also understand the nuances of container orchestration systems, particularly systems which are evolving at a rapid pace, is very hard. Which brings me onto the next point.

It requires serious expertise

When done by experts, the results can be wonderful. But imagine an organisation where perhaps things are not running smoothly with a single monolithic system. What possible reason would there be that things would be any better by increasing the number of systems, which increases the operational complexity?

Yes, with effective automation, monitoring, orchestration and so on, this is all possible. But the challenge is rarely the technology - the challenge is finding people who can use it effectively. These skillsets are currently in very high demand, and may be difficult to find.

Real world systems often have poorly defined boundaries

In all of the examples we used to describe the benefits of microservices, we spoke about independent components. However in many cases components are simply not independent. On paper, certain domains may look bounded, but as you get into the muddy details, you may find that they are more challenging to model than you anticipated.

This is where things can get extremely complex. If your boundaries are actually not well defined, then what happens is that even though theoretically services can be deployed in isolation, you find that due to the inter-dependencies between services, you have to deploy sets of services as a group.

This then means that you need to manage coherent versions of services which are proven and tested when working together, you don’t actually have an independently deployable system, because to deploy a new feature, you need to carefully orchestrate the simultaneous deployment of many services.

The complexities of state are often ignored

In the previous example, I mentioned that a feature deployment may require the simultaneous rollout of many versions of many services in tandem. It is tempting to assume that sensible deployment techniques will mitigate this, for example blue/green deployments (which most service orchestration platforms handle with little effort), or multiple versions of a service being run in parallel, with consuming channels deciding which version to use.

These techniques mitigate a large number of the challenges if the services are stateless. But stateless services are quite frankly, easy to deal with. In fact, if you have stateless services, then I’d be inclined to consider skipping microservices altogether and consider using a serverless model.

In reality, many services require state. An example from our video sharing platform might be the subscription service. A new version of the subscriptions service may store data in the subscriptions database in a different shape. If you are running both services in parallel, you are running the system with two schemas at once. If you do a blue green deployment, and other services depend on data in the new shape, then they must be updated at the same time, and if the subscription service deployment fails and rolls back, they might need to roll back too, with cascading consequences.

Again, it might be tempting to think that with NoSQL databases these issues of schema go away, but they don’t. Databases which don’t enforce schema do not lead to schemaless systems - they just mean that schema tends to be managed at the application level, rather than the database level. The fundamental challenge of understanding the shape of your data, and how it evolves, cannot be eliminated.

The complexitities of communication are often ignored

As you build a large network of services which depend on each other, the liklihood is that there will be a lot of inter-service communication. This leads to a few challenges. Firstly, there are a lot more points at which things can fail. We must expect that network calls will fail, which means when one service calls another, it should expect to have to retry a number of times at the least. Now when a service has to potentially call many services, we end up in a complicated situation.

Imagine a user uploads a video in the video sharing service. We might need to run the upload service, pass data to the transcode service, update subscriptions, update recommendations and so on. All of these calls require a degree of orchestration, if things fail we need to retry.

This retry logic can get hard to manage. Trying to do things synchronously often ends up being untenable, there are too many points of failure. In this case, a more reliable solution is to use asynchronous patterns to handle communication. The challenge here is that asynchronous patterns inherently make a system stateful. As mentioned in the previous point, stateful systems and systems with distributed state are very hard to handle.

When a microservice system uses message queues for intra-service communication, you essentially have a large database (the message queue or broker) glueing the services together. Again, although it might not seem like a challenge at first, schema will come back to bite you. A service at version X might write a message with a certain format, services which depend on this message will also need to be updated when the sending service changes the details of the message it sends.

It is possible to have services which can handle messages in many different formats, but this is hard to manage. Now when deploying new versions of services, you will have times where two different versions of a service may be trying to process messages from the same queue, perhaps even messages sent by different versions of a sending service. This can lead to complicated edge cases. To avoid these edge cases, it may be easier to only allow certain versions of messages to exist, meaning that you need to deploy a set of versions of a set of services as a coherent whole, ensuring messages of older versions are drained appropriately first.

This highlights again that the idea of independent deployments may not hold as expected when you get into the details.

Versioning can be hard

To mitigate the challenges mentioned previously, versioning needs to be very carefully managed. Again, there can be a tendency to assume that following a standard such as semver[4] will solve the problem. It doesn’t. Semver is a sensible convention to use, but you will still have to track the versions of services and APIs which can work together.

This can get very challenging very quickly, and may get to the point where you don’t know which versions of services will actually work properly together.

Managing dependencies in software systems is notoriously hard, whether it is node modules, Java modules, C libraries or whatever. The challenges of conflicts between independent components when consumed by a single entity are very hard to deal with.

These challenges are hard to deal with when the dependencies are static, and can be patched, updated, edited and so on, but if the dependencies are themselves live services, then you may not be able to just update them - you may have to run many versions (with the challenges already described) or bring down the system until it is fixed holistically.

Distributed Transactions

In situations where you need transaction integrity across an operation, microservices can be very painful. Distributed state is hard to deal with, many small units which can fail make orchestrating transactions very hard.

It may be tempting to attempt to avoid the problem by making operations idempotent, offering retry mechanisms and so on, and in many cases this might work. But you may have scenarios where you simply need a transaction to fail or succeed, and never be in an intermediate state. The effort involved in working around this or implementing it in a microservice model may be very high.

Microservices can be monoliths in disguise

Yes, individual services and components may be deployed in isolation, however in most cases you are going to have to be running some kind of orchestration platform, such as Kubernetes. If you are using a managed service, such as Google’s GKE⁴ or Amazon’s EKS⁵, then a large amount of the complexity of managing the cluster is handled for you.

However, if you are managing the cluster yourself, you are managing a large, complicated, mission critical system. Although the individual services may have all of the benefits described earlier, you need to very carefully manage your cluster. Deployments of this system can be hard, updates can be hard, failover can be hard and so on.

In many cases the overall benefits are still there, but it is important not to trivialise or underestimate the additional complexity of managing another big, complex system. Managed services may help, but in many cases these services are nascent (Amazon EKS was only announced at the end of 2017 for example).

Networking Nightmares

A more traditional model of services running on known hosts, with known addresses, has a fairly simple networking setup.

However, when using microservices, generally there will be many services distributed across many nodes, which typically means there’s going to be a much more complicated networking arrangement. There will be load balancing between services, DNS may be more heavily used, virtual networking layers, etc etc, to attempt to ‘hide’ the complexity of this networking.

However, as per Tesler’s Law (or the Law of Conservation of Compexlity), this networking complexity is inherent - when you are finding real, runtime issues in larger scale clusters, it can often be at a very low networking level. These sorts of issues can be very hard to diagnose. I have started tracking some examples at the end of the article, but I think that Tinder’s Migration to Kuberenetes shows this challenge very well.

Overall - the transition is still likely to be for the best, but doesn’t come without some serious challenges at the networking level, which will require some serious expertise to deal with!

The Death of Microservice Madness!

Avoid the madness by making careful and considered decisions. To help out on this I’ve noted a few questions you might want to ask yourself, and what the answers might indicate:

You can download a PDF copy here: microservice-questions.pdf

Final Thoughts: Don’t Confuse Microservices with Architecture

I’ve deliberately avoided the ‘a’ word in this article. But my friend Zoltan made a very good point when proofing this article (which he has contributed to).

There is no microservice architecture. Microservices are just another pattern or implementation of components, nothing more, nothing less. Whether they are present in a system or not does not mean that the architecture of the system is solved.

Microservices relate in many ways more to the technical processes around packaging and operations rather than the intrinsic design of the system. Appropriate boundaries for components continues to be one of the most important challenges in engineering systems.

Regardless of the size of your services, whether they are in Docker containers or not, you will always need to think carefully about how to put a system together. There are no right answers, and there are a lot of options.

I hope you found this article interesting! As always, please do comment below if you have any questions or thoughts. You can also follow some lively discussions on:

Appendix: Further Reading

The following links might be of interest:

Martin Fowler - Bounded Context - Martin’s articles are great, I’d thoroughly recommend this.
Martin Fowler - Microservices - An often recommended introduction to the pattern.
Microservices - Good or Bad? - Björn Frantzén’s thoughts on microservices, after reading this article.
When Not To Do Microservices - Excellent post on the topic from Christian Posta
Sean Hull - 30 questions to ask a serverless fanboy - Interesting thoughts on the challenges of serverless, from a serverless fan!
Dave Kerr - Monoliths to Microservices - Practical tips for CI/CD and DevOps in the Microservice world - A recent conference presentation I did on devops with microservices.
Alexander Yermakov - Microservices without fundamentals - A response to this article, with Alex’s thoughts and counterpoints to the points raised here (see also Microservices as a self sufficient concept)

Please do share anything else you think makes great reading or watching on the topic!

Thanks

Thanks José from campusmvp.es for having the article translated in Spanish - La muerte de la locura de los microservicios en 2018!

Case Studies

Some interesting examples of experiences I am collecting of larger organisations who have made large scale transitions to microservices:

Tinder’s Move to Kubernetes

References

https://trends.google.com/trends/explore?date=today%205-y&q=microservice ↩︎
If you don’t want to miss the article, you can subscribe to the RSS Feed, or follow me on LinkedIn or Twitter. ↩︎
Docker Compose is a good solution, Fuge is very clever, and there is also the option of running orchestration locally as is the case with something like MiniKube. ↩︎
Google Kubernetes Engine, a managed service from Google Cloud Platform for Kubernetes: https://cloud.google.com/kubernetes-engine/ ↩︎
Amazon Elastic Container Services for Kubernetes, a managed service from Amazon Web Services for Kubernetes: https://aws.amazon.com/eks/ ↩︎

Get up and running with OpenShift on AWS

Thu, 02 Feb 2017 07:47:00 +0000

OpenShift is Red Hat’s platform-as-a-service offering for hosting and scaling applications. It’s built on top of Google’s popular Kubernetes system.

Getting up and running with OpenShift Online is straightforward, as it is a cloud hosted solution. Setting up your own cluster is a little more complex, but in this article I’ll show you how to make it fairly painless.

The repo for this project is at: github.com/dwmkerr/terraform-aws-openshift.

Creating the Infrastructure

OpenShift has some fairly specific requirements about what hardware it runs on¹. There’s also DNS to set up, as well as internet access and so on.

All in all, for a bare-bones setup, you’ll need something like this:

Which is (deep breath):

A network
A public subnet, with internet access via a gateway
A master host, which will run the OpenShift master
A pair of node hosts, which will run additional OpenShift nodes
A hosted zone, which allows us to configure DNS
A bastion, which allows us to SSH onto hosts, without directly exposing them
Some kind of basic log aggregation, which I’m using CloudWatch for

This is not a production grade setup, which requires redundant masters and so on, but it provides the basics.

Rather than setting this infrastructure up by hand, this is all scripted with Terraform. To set up the infrastructure, clone the github.com/dwmkerr/terraform-aws-openshift repo:

$ git clone git@github.com:dwmkerr/terraform-aws-openshift
...
Resolving deltas: 100% (37/37), done.

Then use the terraform CLI² to create the infrastructure:

$ cd terraform-aws-openshift/
$ terraform get && terraform apply

You’ll be asked for a region, to deploy the network into, here I’m using us-west-1:

After a few minutes the infrastructure will be set up:

A quick glance at the AWS console shows the new hosts we’ve set up:

The next step is to install OpenShift.

Installing OpenShift

There are a few different ways to install OpenShift, but the one we’ll use is called the ‘advanced installation³’. This essentially involves:

Creating an ‘inventory’, which specifies the hosts OpenShift will be installed on and the installation options
Downloading the advanced installation code
Running the advanced installation Ansible Playbook

To create the inventory, we just run:

sed "s/\${aws_instance.master.public_ip}/$(terraform output master-public_ip)/" inventory.template.cfg > inventory.cfg

This takes our ‘inventory template⁴’ and populates it with the public IP of our master node, which is recorded in a Terraform output variable.

We can then copy the inventory to the bastion:

ssh-add ~/.ssh/id_rsa
scp ./inventory.cfg ec2-user@$(terraform output bastion-public_dns):~

We can again use the Terraform output variables, this time to get the bastion IP. Finally, we pipe our install script to the bastion host:

cat install-from-bastion.sh | ssh -A ec2-user@$(terraform output bastion-public_dns)

There’s a bug which means you might see ansible-playbook: command not found, if so, just run the script again. The install script clones the installation scripts and runs them, using the inventory we’ve provided:

This’ll probably take about 10 minutes to run. And that’s it, OpenShift is installed:

open "https://$(terraform output master-public_dns):8443"

Hit ‘advanced’ and continue, as we’re using a self-signed certificate most browsers will complain:

Enter any username and password (the system is configured to allow anyone to access it by default) and you’ll be presented with the OpenShift console:

As the setup requires three t2.large instances, which are not available on the free plan, you might want to clean up when you are done with:

terraform destroy

Wrapping Up

Hopefully you’ve found this useful, there are more details and references on the README of the github repo:

https://github.com/dwmkerr/terraform-aws-openshift

Comments and feedback are always welcome!

See https://docs.openshift.org/latest/install_config/install/prerequisites.html#system-requirements ↩︎
Use ‘brew install terraform’, full instructions in the README.md ↩︎
See https://docs.openshift.org/latest/install_config/install/advanced_install.html ↩︎
See https://github.com/dwmkerr/terraform-aws-openshift/blob/master/inventory.template.cfg ↩︎

Creating a Resilient Consul Cluster for Docker Microservice Discovery with Terraform and AWS

Mon, 09 Jan 2017 07:10:40 +0000

In this article I’m going to show you how to create a resilient Consul cluster, using Terraform and AWS. We can use this cluster for microservice discovery and management. No prior knowledge of the technologies or patterns is required!

The final code is at github.com/dwmkerr/terraform-consul-cluster. Note that it has evolved somewhat since the time of writing, see the Appendices at the end of the article for details.

Consul, Terraform & AWS

Consul is a technology which enables Service Discovery¹, a pattern which allows services to locate each other via a central authority.

Terraform is a technology which allows us to script the provisioning of infrastructure and systems. This allows us to practice the Infrastructure as Code pattern. The rigour of code control (versioning, history, user access control, diffs, pull requests etc) can be applied to our systems.

And why AWS? We need to create many servers and build a network to see this system in action. We can simulate parts of this locally with tools such as Vagrant, but we can use the arguably most popular² IaaS platfom for this job at essentially zero cost, and learn some valuable skills which are readily applicable to other projects at the same time.

A lot of what we will learn is not really AWS specific - and the Infrastructure as Code pattern which Terraform helps us apply allows us to apply these techniques easily with other providers.

The Goal

The goal is to create a system like this:

In a nutshell:

We have a set of homogenous Consul nodes
The nodes form a cluster and automatically elect a leader
The nodes span more than one availability zone, meaning the system is redundant and can survive the failure of an entire availability zone (i.e. data centre)
The Consul UI is available to view via a gateway
We have two example microservices which register themselves on the cluster, so we can actually see some registered services in the console

As a quick caveat, in reality this setup would typically live in a private subnet, not directly accessible to the outside work except via public facing load balancers. This adds a bit more complexity to the Terraform setup but not much value to the walk-though. A network diagram of how it might look is below, I invite interested readers to try and move to this model as a great exercise to cement the concepts!

Step 1 - Creating our Network

The first logical step is to create the network itself. This means:

The network (in AWS terminology, this is a VPC or Virtual Private Cloud)
The ‘public’ subnet, which defines our IP ranges for hosts
The internet gateway, which provides an entry/exit point for traffic from/to the internet
The firewall rules, which define what traffic can come in and out of the network

All together, that’s this:

Our solution will be made more resilient by ensuring we host our Consul nodes across multiple availability zones³

Creating a VPC and building a subnet is fairly trivial if you have done some network setup before or spent much time working with AWS, if not, you may be a little lost already. There’s a good course on Udemy⁴ which will take you through the process of setting up a VPC which I recommend if you are interested in this, as it is quite hands on. It’ll also show you how to build a more ‘realistic’ network, which also contains a private subnet and NAT, but that’s beyond the scope of this write-up. Instead, I’ll take you through the big parts.

The Network

We’re using AWS, we need to create a VPC. A VPC is a Virtual Private Cloud. The key thing is that it is isolated. Things you create in this network will be able to talk to each other if you let them, but cannot communicate with the outside world, unless you specifically create the parts needed for them to do so.

A private network is probably something you regularly use if you work in a company⁵. Most companies have their own internal network - when you use a computer on that network it can talk to other company computers (such as the company mail server). When you are off that network, you might not be able to access your company email (unless it is publicly available, like gmail, or over a VPN [and by accessing a VPN, you are actually joining the network again, albeit remotely]).

Perhaps the most immediately obvious part of a VPC is that you control the IP addresses. You specify the range of IP addresses which are available to give to machines on the network. When a machine joins, it is given an IP in that range. I’m not going to go into too much detail here, if you are interested let me know and I’ll write up an article on VPCs in detail!

Here’s how I’d suggest scripting AWS infrastructure with Terraform if you haven’t done this before.

Use the AWS console to create what you want
Search the Terraform documentation for the entity you want to create (e.g. VPC), script the component and apply the provisioning
Compare the hand-made VPC to the script-made VPC, if the two are the same, you are done
If the two are different, check the documentation and try again

Ensure you have an AWS account, and note your Secret Key and Access Key. We’ll need these to remotely control it. Here’s the terraform script to create a VPC:

// Setup the core provider information.
provider "aws" {
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
region = "${var.region}"
}
// Define the VPC.
resource "aws_vpc" "consul-cluster" {
cidr_block = "10.0.0.0/16" // i.e. 10.0.0.0 to 10.0.255.255
enable_dns_hostnames = true
tags {
Name = "Consul Cluster VPC"
Project = "consul-cluster"
}
}

This script uses Terraform Variables, such as var.access_key, which we keep in a variables.tf file. Terraform will use the default values defined in the file if they are present, or ask the user to supply them. Let’s build the network:

terraform apply

After supplying the values for the variables, Terraform will provision the network, using the AWS SDK internally.

You’ll see lots of info about what it is creating, then a success message.

The Public Subnet

You don’t put hosts directly into a VPC, they need to go into a structure called a ‘subnet’, which is a part of a VPC. Subnets get their own subset of the VPC’s available IP addresses, which you specify.

Subnets are used to build zones in a network. Why would you need this? Typically it is to manage security. You might have a ‘public zone’ in which all hosts can be accessed from the internet, and a ‘private zone’ which is inaccessible directly (and therefore a better location for hosts with sensitive data). You might have an ‘operator’ zone, which only sysadmins can access, but they can use to get diagnostic information.

Here’s a common subnet layout for multi-tiered applications:

The defining characteristics of zones is that they are used to create boundaries to isolate hosts. These boundaries are normally secured by firewalls, traversed via gateways or NATs etc. We’re going to create two public subnets, one in each of the availability zones⁵:

// Create a public subnet for each AZ.
resource "aws_subnet" "public-a" {
vpc_id = "${aws_vpc.consul-cluster.id}"
cidr_block = "10.0.1.0/24" // i.e. 10.0.1.0 to 10.0.1.255
availability_zone = "ap-southeast-1a"
map_public_ip_on_launch = true
}
resource "aws_subnet" "public-b" {
vpc_id = "${aws_vpc.consul-cluster.id}"
cidr_block = "10.0.2.0/24" // i.e. 10.0.2.0 to 10.0.1.255
availability_zone = "ap-southeast-1b"
map_public_ip_on_launch = true
}

With Terraform, resources can depend on each other. In this case, the subnets need to reference the ID of the VPC we want to place them in (so we use aws_vpc.consul-cluster.id).

The Internet Gateway, Route Tables and Security Groups

The final parts of the network you can see in the ./infrastructure/network.tf script. These are the Internet Gateway, Route Table and Security Group resources. Essentially they are for controlling access between hosts and the internet. AWS have a good guide if you are not familiar with these resources; they don’t add much to the article so I’ll leave you to explore on your own.

That’s it for the network, we now have the following structure:

If you want to see the code as it stands now, check the Step 1 branch. Now we need to look at creating the hosts to install Consul on.

Step 2 - Creating the Consul Hosts

The Consul documentation recommends running in a cluster or 3 or 5 nodes⁶. We want to set up a system which is self-healing - if we lose a node, we want to create a new one.

Enter Auto-Scaling Groups. Auto-scaling groups allow us to define a template for an instance, and ask AWS to make sure there are always a certain number of these instances. If we lose an instance, a new one will be created to keep the group at the correct size⁷.

So we now need to create:

A ‘Launch Configuration’ which determines what instances our Auto-scaling Group creates
A ‘user data script’ which runs on newly created instances, which must install and start Consul
An Auto-scaling group, configured to run five instances across the two public subnets
A load balancer, configured to pass incoming requests for the Consul Admin console to the nodes

Or visually:

Let’s get to it.

The Launch Configuration & Auto-scaling Group

The Launch Configuration will define the characteristics of our instances and the auto-scaling group determines the size of our cluster:

// Launch configuration for the consul cluster auto-scaling group.
resource "aws_launch_configuration" "consul-cluster-lc" {
name_prefix = "consul-node-"
image_id = "${lookup(var.ami_ecs_optimised, var.region)}"
instance_type = "t2.micro"
security_groups = ["${aws_security_group.consul-cluster-vpc.id}"]
lifecycle {
create_before_destroy = true
}
}
// Auto-scaling group for our cluster.
resource "aws_autoscaling_group" "consul-cluster-asg" {
name = "consul-asg"
launch_configuration = "${aws_launch_configuration.consul-cluster-lc.name}"
min_size = 5
max_size = 5
vpc_zone_identifier = [
"${aws_subnet.public-a.id}",
"${aws_subnet.public-b.id}"
]
lifecycle {
create_before_destroy = true
}
}

A few key things to note:

I have omitted the tag properties in the scripts for brevity
The ‘image’ for the launch configuration is looked up based on the region we’ve specified - we’re a basic linux image⁸
We are using micro instances, which are free-tier eligible
The auto-scaling group spans both availability zones.

Once we run terraform apply, we’ll see our auto-scaling group, which references the new launch configuration and works over multiple availability zones:

We can also see the new instances:

These instances don’t do much yet though, we’ve not installed Docker or Consul.

Installing Consul and Accessing the Admin Interface

To set up our instances we use a ‘userdata’ script’ A userdata runs once when an instance is created. We can create a script in our repository, and reference it in our Terraform files.

We add a new file called consul-node.sh to a files folder. This script installs Docker and runs Consul:

yum install -y docker
usermod -a -G docker ec2-user
service docker start

# Get my IP address.
IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
echo "Instance IP is: $IP"

# Start the Consul server.
docker run -d --net=host \
 --name=consul \
 consul agent -server -ui \
 -bind="$IP" \
 -client="0.0.0.0" \
 -bootstrap-expect="1"

Here’s a breakdown of what we’re doing:

Install Docker. These scripts run as root, so we add the ec2-user to the Docker group, meaning when we log in later on via SSH, we can run Docker
Get our IP address. AWS provide a magic address (169.254.169.254) which lets you query data about your instance, see Instance Metadata & User Metadata
Run the Consul docker image in server mode, with the UI enabled, expecting only one instance

The actual scripts contains more! Getting userdata scripts right, testing and debugging them is tricky. See how I do it in detail in Appendix 1: Logging.

Now we need to tell Terraform to include this script as part of the instance metadata. Here’s how we do that:

resource "aws_launch_configuration" "consul-cluster-lc" {
/// ...add the line below....
user_data = "${file("files/consul-node.sh")}"
}

When Consul is running with the -ui option, it provides an admin UI. You can try it by running Consul locally with docker run -p8500:8500 consul and navigating to http://localhost:8500/ui.

We can install a load balancer in front of our auto-scaling group, to automatically forward incoming traffic to a host. Here’s the config:

resource "aws_elb" "consul-lb" {
name = "consul-lb-a"
security_groups = [
"${aws_security_group.consul-cluster-vpc.id}",
"${aws_security_group.web.id}"
]
subnets = [
"${aws_subnet.public-a.id}",
"${aws_subnet.public-b.id}"
]
listener {
instance_port = 8500
instance_protocol = "http"
lb_port = 80
lb_protocol = "http"
}
health_check {
healthy_threshold = 2
unhealthy_threshold = 2
timeout = 3
target = "HTTP:8500/ui/"
interval = 30
}
}

Blow-by-blow:

Create a load balancer, with the same security groups as the rest of the VPC, but also a security group which allows web access
Point to two subnets first subnet
Forward HTTP 8500 traffic
Configure a healthcheck⁹

The final change we make is to add an outputs.tf file, which lists all of the properties Terraform knows about which we want to save. All it includes is:

output "consul-dns" {
value = "${aws_elb.consul-lb.dns_name}"
}

When we finally run terraform apply, we see the public DNS of our load balancer:

And running in a browser on port 8500 we see the Consul admin interface:

Every time we refresh we will likely see a different node. We’ve actually created five clusters each of one node - what we now need to do is connect them all together into a single cluster of five nodes.

If you want to see the code as it stands now, check the Step 2 branch.

Step 3 - Creating the Cluster

Creating the cluster is now not too much of a challenge. We will update the userdata script to tell the consul process we are expecting 5 nodes (via the bootstrap-expect flag.

Here’s the updated script:

# Get my IP address.
IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
echo "Instance IP is: $IP"
# Start the Consul server.
docker run -d --net=host \
--name=consul \
consul agent -server -ui \
-bind="$IP" \
-client="0.0.0.0" \
-bootstrap-expect="5"

The problem is this won’t work… We need to tell each node the address of another server in the cluster. For example, if we start five nodes, we should tell nodes 2-5 the address of node 1, so that the nodes can discover each other.

The challenge is how do we get the IP of node 1? The IP addresses are determined by the network, we don’t preset them so cannot hard code them. Also, we can expect nodes to occasionally die and get recreated, so the IP addresses of nodes will in fact change over time.

Getting the IP addresses of nodes in the cluster

There’s a nice trick we can use here. We can ask AWS to give us the IP addresses of each host in the auto-scaling group. If we tell each node the addresses of the other nodes, then they will elect a leader themselves¹⁰.

There are a couple of things we need to do to get this right. First, update the userdata script to provide the IPs of other nodes when we’re starting up, then update the role of our nodes so that they have permissions to use the APIs we’re going to call.

Getting the Cluster IPs

This is actually fairly straightforward. We update our userdata script to the below:

# A few variables we will refer to later...
ASG_NAME=consul-asg
REGION=ap-southeast-1
EXPECTED_SIZE=5

# Return the id of each instance in the cluster.
function cluster-instance-ids {
 # Grab every line which contains 'InstanceId', cut on double quotes and grab the ID:
 # "InstanceId": "i-example123"
 #....^..........^..^.....#4.....^...
 aws --region="$REGION" autoscaling describe-auto-scaling-groups \
 --auto-scaling-group-name $ASG_NAME \
 | grep InstanceId \
 | cut -d '"' -f4
}

# Return the private IP of each instance in the cluster.
function cluster-ips {
 for id in $(cluster-instance-ids)
 do
 aws --region="$REGION" ec2 describe-instances \
 --query="Reservations[].Instances[].[PrivateIpAddress]" \
 --output="text" \
 --instance-ids="$id"
 done
}

# Wait until we have as many cluster instances as we are expecting.
while COUNT=$(cluster-instance-ids | wc -l) && [ "$COUNT" -lt "$EXPECTED_SIZE" ]
do
 echo "$COUNT instances in the cluster, waiting for $EXPECTED_SIZE instances to warm up..."
 sleep 1
done

# Get my IP address, all IPs in the cluster, then just the 'other' IPs...
IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
mapfile -t ALL_IPS < <(cluster-ips)
OTHER_IPS=( ${ALL_IPS[@]/${IP}}/} )
echo "Instance IP is: $IP, Cluster IPs are: ${CLUSTER_IPS[@]}, Other IPs are: ${OTHER_IPS[@]}"

# Start the Consul server.
docker run -d --net=host \
 --name=consul \
 consul agent -server -ui \
 -bind="$IP" \
 -retry-join="${OTHER_IPS[0]}" -retry-join="${OTHER_IPS[1]}" \
 -retry-join="${OTHER_IPS[2]}" -retry-join="${OTHER_IPS[3]}" \
 -bootstrap-expect="$EXPECTED_SIZE"

Right, here’s what’s going on:

We create a few variables we’ll use repeatedly
We create a cluster-instance-ids function which returns the ID of each instance in the auto-scaling group
We create a cluster-ips function which returns the private IP address of each instance in the cluster.
We wait until the auto-scaling group has our expected number of instances (it can take a while for them all to be created)
We get the 5 IP addresses
We remove our IP from the array, leaving us with the IPs of the other nodes
We start the Consul agent in server mode, expecting 5 nodes and offering the IP of each other agent

The problem is, if we try to run the script we will fail, because calling the AWS APIs requires some permissions we don’t have. Let’s fix that.

Creating a Role for our nodes

Our nodes now have a few special requirements. They need to be able to query the details of an auto-scaling group and get the IP of an instance¹¹.

We will need to create a policy which describes the permissions we need, create a role, attach the policy to the role and then ensure our instances are assigned the correct role. This is consul-node-role.tf file:

// This policy allows an instance to discover a consul cluster leader.
resource "aws_iam_policy" "leader-discovery" {
name = "consul-node-leader-discovery"
path = "/"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1468377974000",
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeAutoScalingGroups",
"ec2:DescribeInstances"
],
"Resource": [
"*"
]
}
]
}
EOF
}
// Create a role which consul instances will assume.
// This role has a policy saying it can be assumed by ec2
// instances.
resource "aws_iam_role" "consul-instance-role" {
name = "consul-instance-role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
// Attach the policy to the role.
resource "aws_iam_policy_attachment" "consul-instance-leader-discovery" {
name = "consul-instance-leader-discovery"
roles = ["${aws_iam_role.consul-instance-role.name}"]
policy_arn = "${aws_iam_policy.leader-discovery.arn}"
}
// Create a instance profile for the role.
resource "aws_iam_instance_profile" "consul-instance-profile" {
name = "consul-instance-profile"
roles = ["${aws_iam_role.consul-instance-role.name}"]
}

Terraform is a little verbose here! Finally, we update our launch configuration to ensure that the instances assume this role.

resource "aws_launch_configuration" "consul-cluster-lc" {
// Add this line!!
iam_instance_profile = "${aws_iam_instance_profile.consul-instance-profile.id}"
}
}

Let’s create the cluster again, with terraform apply. When we log into the UI we should now see a cluster containing all five nodes:

This code is all in the Step 3 branch.

If you are familiar with Consul, this may be all you need. If not, you might be interested in seeing how we actually create a new instance to host a service, register it with Consul and query its address.

Step 4 - Adding a Microservice

I’ve created a docker image for as simple a microservice as you can get. It returns a quote from Futurama’s Zapp Brannigan. The image is tagged as dwmkerr/zapp-service.

On a new EC2 instance, running in either subnet, with the same roles as the Consul nodes, we run the following commands:

# Install Docker
sudo su
yum update -y
yum install -y docker
service docker start
# Get my IP and the IP of any node in the server cluster.
IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
NODE_ID=$(aws --region="ap-southeast-1" autoscaling describe-auto-scaling-groups --auto-scaling-group-name "consul-asg" \
| grep InstanceId \
| cut -d '"' -f4 \
| head -1)
NODE_IP=$(aws --region="ap-southeast-1" ec2 describe-instances \
--query="Reservations[].Instances[].[PrivateIpAddress]" \
--output="text" \
--instance-ids="$NODE_ID")
# Run the consul agent.
docker run -d --net=host \
consul agent \
-bind="$IP" \
-join=$NODE_IP
# Run registrator - any Docker images will then be auto registered.
docker run -d \
--name=registrator \
--net=host \
--volume=/var/run/docker.sock:/tmp/docker.sock \
gliderlabs/registrator:latest \
consul://localhost:8500
# Run the example microservice - registrator will take care of letting consul know.
docker run -d -p 5000:5000 dwmkerr/zapp-service

What’s going on here?

We grab our own IP address and the IP address of the first instance we find in the server cluster, using the same tricks as before
We run the Consul agent - telling it the IP to use to join the cluster
We run Registrator, a handy utility which will automatically register any new services we run to Consul
We run a goofy sample microservice (which registrator will register for us)

Now we can check the Consul UI:

And there we have it. Our new node joins the cluster (as a client), we can register a new service with Consul.

We can call this service from any node in the subnet, seeing output like the below:

In this example, I used a DNS SRV query to ask where the zapp-service is, was told it was at 10.0.2.158 on port 5000, then called the service, receiving a response. I can discover any service using this method, from any node. As services are added, removed, moved etc, I can ask Consul for accurate information on where to find them.

Check the Step 4 branch to see the code in its current state.

Step 5 - Spanner Throwing

We can now try to throw some spanners in the works, to see how resilient the system is.

According to the Deployment Table from the Consul documentation, a cluster of five nodes means we have a quorum of three nodes (i.e. a minimum of three nodes are needed for a working system). This means we can tolerate the failure of two nodes.

The easiest way to test this is to simply manually kill two nodes:

If we pick two random nodes, as above, and terminate them, we see the cluster determines that we have two failed nodes but will still function (if one was the leader, a new leader will be automatically elected):

What’s nice about this setup is that no manual action is needed to recover. Our load balancer will notice the nodes are unhealthy and stop forwarding traffic. Our auto-scaling group will see the nodes have terminated and create two new ones, which will join the cluster in the same way as the original nodes. Once they join, the load balancer will find them healthy and bring them back into rotation.

We can see from the load balancer monitoring that it notices we have unhealthy nodes and also notices when new ones come into service:

A quick check of the admin dashboard shows we now have a recovered system, with five healthy nodes:

The nodes which were terminated are still listed as failing. After 72 hours Consul will stop trying to periodically reconnect to these nodes and completely remove them¹².

Wrapping Up

Hopefully this should provide a good starting point to think about building your own resilient and robust systems for services like Consul.

Interesting areas to look into to extend the project would be:

Setting up alerts so that if we lose more than one node, we are informed
Automating resilience tests by programatically bringing down servers and monitoring how long it takes the system to return to five nodes
Instead of using a userdata script to set up a node, bake it into a new custom AMI with Packer
Adding alerts for if we lose three of more nodes, which always requires manual intervention (see Outage Recovery)

As always, any questions or comments are welcome! All code is available at github.com/dwmkerr/terraform-consul-cluster.

Appendix 1: Logging

Small typos or mistakes in the userdata script are almost impossible to effectively diagnose. The scripts were actually built in the following way:

Draft a script on my local machine which configures script logging and CloudWatch¹³
Spin up a new EC2 instance manually
SSH onto the instance, and run the script line by line until I’m sure it’s right
Ensure the logs are forwarded to CloudWatch, then add the more complex features and repeatedly test

I’ve included CloudWatch logging in the code. In this write-up I’ve omitted this code as it is purely for diagnostics and doesn’t contribute to the main topic. The setup is in the consul-node.sh and consul-node-role.tf files.

If you want more details, let me know, or just check the code. I would heartily recommend setting up logging like this for all but the most straightforward projects:

Being able to diagnose issues like this is vital when working with distributed systems which may be generating many log files.

Appendix 2: Modularisaton

I got some a great PR from arehmandev which modularises the code. This makes it more reusable and cleans up the structure significantly. If you want to see the before/after, check the original PR at https://github.com/dwmkerr/terraform-consul-cluster/pull/4.

Footnotes

Further Reading

This kind of pattern is critical in the world of microservices, where many small services will be running on a cluster. Services may die, due to errors or failing hosts, and be recreated on new hosts. Their IPs and ports may be ephemeral.It is essential that the system as a whole has a registry of where each service lives and how to access it. Such a registry must be resilient, as it is an essential part of the system. ↩︎
Most popular is a fairly loose term. Well ranked by Gartner and anecdotally with the largest infrastructure footprint. https://www.gartner.com/doc/reprints?id=1-2G2O5FC&ct=150519&st=sb ↩︎
This is AWS parlance again. An availabilty zone is an isolated datacenter. Theoretically, spreading nodes across AZs will increase resilience as it is less likely to have catastrophic failures or outages across multiple zones. ↩︎
I don’t get money from Udemy or anyone else for writing anything on this blog. All opinions are purely my own and influenced by my own experience, not sponsorship. Your milage may vary (yada yada) but I found the course quite good: https://www.udemy.com/aws-certified-solutions-architect-associate/. ↩︎
For more expert readers that may sound horribly patronising, I don’t mean it to be. For many less experienced technologists the basics of networking might be more unfamiliar! ↩︎ ↩︎
See https://www.consul.io/docs/internals/consensus.html. ↩︎
A common pattern is to actually make the group size dynamic, responding to events. For example, we could have a group of servers which increases in size if the average CPU load of the hosts stays above 80% for five minutes, and scales down if it goes below 10% for ten minutes. This is more common for app and web servers and not needed for our system. ↩︎
Specifically, the current latest Amazon Linux AMI. ↩︎
Check the admin UI every 30 seconds, more than 3 seconds indicates a timeout and failure. Two failures in a row means an unhealthy host, which will be destroyed, two successes in a row for a new host means healthy, which means it will receive traffic. ↩︎
This is a fairly sophisticated topic in itself, see Consul - Consensus Protocol for details. ↩︎
In fact, we actually have more permissions required, because in the ‘real’ code we also have logs forwarded to CloudWatch. ↩︎
These nodes can be removed manually, see Consul Force Leave. ↩︎
Amazon’s service for managing and aggregating logs ↩︎

Simple Continuous Integration for Docker Images

Thu, 03 Nov 2016 05:14:35 +0000

In this article I’m going to demonstrate a few tips and tricks which can make your life easier when you are building or maintaining Dockerfiles.

The need for a Build Pipeline

Do we really need any kind of continuous integration or build pipeline for Dockerfiles?

There will be cases when the answer is no. However, if the answer to any of the following questions is ‘yes’, it might be worth considering:

Do you want others to be able to contribute to the Dockerfile, perhaps changing the image over time?
Are there specific functionalities in your Dockerfiles which could break if altered?
Do you expect to need to release updates to your Dockerfile?

Essentially, if we are looking at providing some kind of automated quality assurance and automation around building and releasing, then a build pipeline is not a bad idea.

A simple Build Pipeline

Here’s what a simple build pipeline could look like. This example is for a Docker Image I just created for local DynamoDB development - dwmkerr/docker-dynamodb:

Let’s dissect what we’ve got here.

The Dockerfile

This is the main ‘code’ of the project if you like. The Dockerfile is the recipe for the image we create.

The Continuous Integration Service

In this case, I am using CircleCI, however the approach described would work fine with most CI systems (such as Jenkins, TravisCI and TeamCity). There is an option to use the Docker Hub Automated Builds, but I’ve found this doesn’t give the flexibility I need (see Why not Docker Hub Automated Builds).

Essentially the CI service needs to offer the option to have three distinct steps in the pipeline, each of which must pass for process to proceed:

Build
Test
Deploy

The Build

We can build with tools, script files, whatever. At the moment, I am leaning towards makefiles. Normally I only need a few lines of shell script to do a build - anything more complex and the makefile can call a shell script. See also Why Makefiles?

Here’s what it might look like:

build:
docker build -t dwmkerr/dynamodb:latest .
ifndef BUILD_NUM
$(warning No build number is defined, skipping build number tag.)
else
docker build -t dwmkerr/dynamodb:$(BUILD_NUM) .
endif

This command just builds the Dockerfile and tags it as dwmkerr/dynamodb:lastest. If a BUILD_NUM variable is present, we also create the tag dwmkerr/dynamodb:BUILD_NUM. This means if we want to deploy to a service such as Amazon ECS we can push a specific build by referring to the image with that tag.

The Tests

Again I’m relying on make. I just want to be able to run make test - if zero is returned I’m happy. If not, the pipeline should stop and I’ll check the output. Here’s my test command:

test: build
./test/basics.test.sh
./test/ephemeral.test.sh
./test/persistent.test.sh

Not a thing of beauty, but it works. These scripts I’ll discuss a little bit later on, in the delightly titled What are these test scripts section.

For CircleCI, this is enough to have the main part of our pipeline. Here’s how the circle.yml file looks at this stage:

machine:
services:
- docker
environment:
# Set the build number, used in makefiles.
BUILD_NUM: $CIRCLE_BUILD_NUM
test:
override:
- make test

(Actually there’s a couple of other bits but they’re just to make sure circle uses the right version of Docker, see the full circle.yml file here).

The Deployments

Deployments are trivial as all we need to do is push to the Docker Hub. The make deploy command looks-a like this:

deploy:
docker push dwmkerr/dynamodb:latest
ifndef BUILD_NUM
$(warning No build number is defined, skipping push of build number tag.)
else
docker push dwmkerr/dynamodb:$(BUILD_NUM)
endif

We’re pushing the latest tag and BUILD_NUM tag if present. To add this to the CircleCI pipeline, we just add the following to circle.yml:

deployment:
master:
branch: master
commands:
- docker login -e $DOCKER_EMAIL -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
- make deploy

If we have a push to master, we log in to Docker (using environment variables I configure in the CircleCI UI) and then run make deploy to push our images.

That’s It

That’s about it. This is a pretty simple approach, you can see it in action at:

github.com/dwmkerr/docker-dynamodb

The rest of this post is a bit of a deep dive into some specific areas I found interesting.

Appendix 1: Why not Docker Hub Automated Builds?

There are automated builds available in the Docker Hub:

I’m not using this feauture at the moment, here’s a brief roundup of what I think are the current pros and cons:

Pros

You don’t have to goof around installing Docker on a CI platform.
It allows you to update the description of your Docker image automatically, from the GitHub README.md.
It allows you to associate the image with a specific GitHub repo (rather than just linking from the image description).
Branch management - allowing tags to be built for specific branches.

Cons

It doesn’t seem to support any kind of configurable gating, such as a running a test command prior to deploying.
It doesn’t seem to support any kind of triggering of downstream processes, such as updating environments, sending notifications or whatever.

The lack of ability to perform tests on the image before deploying it why I’m currently not using the service.

By doing the testing in a CI system for every pull request and only merging PRs where the tests pass we could mitigate the risk here. This service is worth watching as I’m sure it will evolve quickly.

Appendix 2: Why Makefiles?

I started coding with a commandline compiler in DOS. When I used my first GUI (Borland Turbo C++) it felt like a huge leap:

Later on I moved onto Microsoft Visual C++ 4.2:

And you cannot imagine the excitement when I got my boxed edition of Visual Studio .NET:

Wow!

Anyway, I digress. GNU make was invented by Leonardo Da Vinci in 1473 to allow you to build something from the commandline, using a fairly consistent syntax.

It is near ubiquitous on *nix systems. I am increasingly using it as an ’entry point’ to builds, as I use variety of languages and platforms. Being able to know that most of the time:

make build
make test

Will build and test something is convenient. Makefiles actually are not that great to work with (see this, this and this). I’ve found as long as you keep the commands simple, they’re OK. For anything really complex, I normally have a scripts/ folder, but call the scripts from the makefile, so that there’s still a simple entrypoint.

I’m not entirely sold on makefiles, but they tend to be my default at the moment if I know I’m going to use the commandline for builds (for example, in Java projects I’ll often write a makefile to call Maven or Gradle).

For things like Node.js, where you have commands like npm test or npm run xyz I still sometimes use makefiles, using npm for day-to-day dev tests (npm start) and make if it’s something more complex (e.g. make deploy-sit to deploy to an SIT environment).

Appendix 3: What are these test scripts?

You may have noticed:

test: build
./test/basics.test.sh
./test/ephemeral.test.sh
./test/persistent.test.sh

What’s going on here?

My Docker image is just a wrapper around Amazon’s Local DynamoDB tool. I don’t really need to test that tool. But what I wanted to test was the capabilities which lie at the intersection between ’native’ Docker and ’native’ DynamoDB.

For example, I know Docker supports volume mapping. I know DynamoDB supports using a data directory, to allow persistent between runs. I want to test I can combine Docker volume mapping and the DynamoDB data directory features. I know Docker images should default to being ephemeral, I want to test this holds true by default for my image.

Testing Docker is a little hard - I want to test that I can run containers, start, stop, check state before and after and so on. This is essentially an integration test, it can be tricky to make it truly isolated and deterministic.

I’ve given it my best go with these scripts. Here’s an example for the ’ephemeral’ test, where I’m trying to assert that if I run a container, create a table, stop the container and run a new one, I no longer have the table. Here’s the test:

# Bomb if anything fails.
set -e

# Kill any running dynamodb containers.
echo "Cleaning up old containers..."
docker ps -a | grep dwmkerr/dynamodb | awk '{print $1}' | xargs docker rm -f || true

# Run the container.
echo "Checking we can run the container..."
ID=$(docker run -d -p 8000:8000 dwmkerr/dynamodb)
sleep 2

# Create a table.
aws dynamodb --endpoint-url http://localhost:8000 --region us-east-1 \
 create-table \
 --table-name Supervillains \
 --attribute-definitions AttributeName=name,AttributeType=S \
 --key-schema AttributeName=name,KeyType=HASH \
 --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

# Clean up the container. On CircleCI the FS is BTRFS, so this might fail...
echo "Stopping and restarting..."
docker stop $ID && docker rm $ID || true
ID=$(docker run -d -p 8000:8000 dwmkerr/dynamodb)
sleep 2

# List the tables - there shouldn't be any!
COUNT=$(aws dynamodb --endpoint-url http://localhost:8000 --region us-east-1 \
 list-tables \
 | jq '.TableNames | length')
if [ $COUNT -ne "0" ]; then
 echo "Expected to find no tables, found $COUNT..."
 exit 1
fi

It’s a bit dirty - it removes containers from the host, changes things and so on. But it works.

I did experiment with running these tests in a container, which has the benefit of giving you a clean host to start with, which you can throw away after each test.

I had to give up after a little while due to time constraints, but will probably revisit this process. The benefits of running these integration tests in a container is that we get a degree of isolation from the host.

If anyone is interested, my attempts so far are on this RFC Pull Request. Feel free to jump in!

Run Amazon DynamoDB locally with Docker

Thu, 27 Oct 2016 08:06:00 +0000

tl;dr: Run DynamoDB locally using Docker:

docker run -d -p 8000:8000 dwmkerr/dynamodb

Try it out by opening the shell, localhost:8000/shell:

That’s all there is to it!

DynamoDB

Amazon DynamoDB is a NoSQL database-as-a-service, which provides a flexible and convenient repository for your services.

Building applications which use DynamoDB is straightforward, there are APIs and clients for many languages and platforms.

One common requirement is to be able to run a local version of DynamoDB, for testing and development purposes. To do this, you need to:

Hit the DynamoDB Local documentation page
Download an archive
Extract it to a sensible location
Run the extracted JAR, perhaps passing in some options

This can be a little cumbersome if you regularly use DynamoDB, so here’s a easier way:

docker run -p 8000:8000 dwmkerr/dynamodb

The dwmkerr/dynamodb image runs the JAR in a container, exposing the database on port 8000 by default.

You can see the image on the Docker Hub and the source code at github.com/dwmkerr/docker-dynamodb.

Customising DynamoDB

You can pass any of the documented commandline flags to DynamoDB. There are instructions on the GitHub page. Here’s an example of how you can pass in a data directory, which allows DynamoDB data to be persisted after restarting a container (the image is ephemeral by default, as per Dockerfile best practices).

docker run -d -p 8000:8000 -v /tmp/data:/data/ dwmkerr/dynamodb -dbPath /data/

Running DynamoDB in a container gives an extra degree of flexibility and can speed up your workflow too!

Testing the Docker for Mac Beta

Fri, 03 Jun 2016 10:45:24 +0000

I’ve finally had a chance to install the new Docker for Mac Beta and give it a whirl. In this article I’m going to talk a bit about how Docker works, the challenges of running Docker on a Mac or Windows and how the new Beta helps.

Below: The welcome message for the new Docker for Mac app

So What is Docker for Mac?

If you don’t know what Docker is, check out my article Learn Docker by Building a Microservice or the lovely What is Docker page from the docs.

You may be aware that Docker creates processes in isolated containers using some key Linux technologies which allow for low-level isolation (such as namespaces and cgroups¹).

This is described in detail on the Understand the Docker Architecture page, but essentially means we can do this:

Here I have:

My machine, called Dave-Ubuntu, which is running Ubuntu², with a local IP 192.168.0.1.
The docker executable, which I use to issue commands to…
…the Docker Host, which runs the docker daemon, which actually does the work of starting/stopping/building containers and so on.
Some containers in the Docker Host - one is based on a MySQL image and has a DB, one is based on a Node.js image and is running an app.

The Docker host is actually my machine - I can connect using the loopback IP 127.0.0.1 (i.e. localhost). The containers also have the IP of the host. If I want to create and connect to a MySQL DB from my machine, I just type:

docker run -d -e MYSQL_ROOT_PASSWORD=123 -p 3306:3306 mysql
mysql -uroot -p123 -h127.0.0.1
> show databases;
> ...etc...
> exit;

The container was created on my machine (in the host) and addressable using my loopback IP.

So What?

This is all great, but things get a little harder on a Mac or Windows. MacOS and the Windows OS don’t have the same kernel level support for process isolation, control groups and so on, so the Docker Host cannot run on these operating systems. Instead, an extra layer and component is introduced:

What’s new?

Oracle VirtualBox has been installed to create and manage virtual machines.
A virtual machine running Linux (called in this case a ‘docker machine’) called ‘default’ has been created (by convention with the IP 192.168.99.100).
This virtual machine runs Linux, so can perfectly happily act as the docker host.
The docker host is still addressable as 127.0.0.1 - from the virtual machine - from the outside world (i.e. my Mac) I have to use the virtual machine IP.

So this is how Docker works on a Mac or on Windows. Things are made seemless where possible, for example, all of the required components are installed when you install the Docker Toolbox.

So What?

Well the problem here is that one of the big benefits of using docker is that it allows us to create development environments which are much closer to production environments (at least from a software point of view).

This kind of breaks down if we are doing development on a Mac or on Windows - because we have introduced an additional component which is simply not going to be present in our production environment. What are the problems?

1. Localhost vs docker-machine IP

Docker helps us be a lot more agnostic to our development box, but if I’m writing about how to interact with docker containers there’s a problem:

docker run -d -p 8080:8080 my-app-server
curl http://localhost:8080/some-api-call

This works on a Linux machine - it does not work on a Mac or Windows. On a Mac I need to run something like:

docker run -d -p 8080:8080 my-app-server
curl http://$(docker-machine ip default)/some-api-call

This will not work on Linux or Windows.

Is this a big deal? Actually, kind of. What if I have an integration test which spins up some containers and runs calls against them - the test has to know about the execution environment. That’s a pain. An alternative is to run tests in a container and link them with something like docker-compose, but this is not ideal.

2. Terminal Hassle

If I open a terminal and check to see what containers are running:

docker ps

I’ll see nothing. If I try to run a container:

docker run -it mongo

I’ll get an error - because my docker instance cannot communicate with the host. I need to use a specially set up terminal to tell it to connect to the VM.

Again, the Docker Toolkit is set up to try and make things easy. If I install the toolkit I can run an app called Docker Quickstart Terminal:

And this will open a terminal where I can use these commands. It will also start the docker machine VM if it has to. It’s even smart enough to recognise if I have multiple terminal apps, such as iTerm, and ask which one I want to use.

This problem is - this doesn’t always work smoothly. Sometimes it will seem that the machine has started but will still not accept commands. Typically a restart is needed in this scenario.

Also, it’s an interruption. If you are running a terminal already and want to issue a quick command, it will fail, unless it was a terminal started with the Docker Quickstart app.

3. inotify - In Container Development

If you recognise the term, you probably know the issue. If not, a little explanation is necessary.

As you get more and more familiar with Docker, you will probably find that you are spending more and more time testing, building then running your image in a container. In fact, you might be changing a single code file and using the container as the dev test server on your machine.

This fast gets painful - the container image takes time to build and slows down the development cycle. There’s a great technique in this scenario: In Container Development.

In container development is pretty much what it sounds like. Instead of editing your code on your machine, building an image and creating a container to debug, you simply create the container with what you need, mount your code in to the container and run all of your development tooling from inside the container:

In this diagram, I have my code locally on my machine. I have built a container which runs nodemon, watching a directory on the container. That directory is actually just a volume containing my code which I have mounted into my container.

This is a really nice technique - I can still code locally, but as I make changes, nodemon serves up my new content.

This specific example applies to Node.js, but can be applied to many scenarios.

The problem is that many watcher tools like nodemon use a kernel subsystem called inotify³ to get notifications when files change. But inotify doesn’t work on virtualbox⁴. This means that this technique isn’t supported for Mac or Windows. There are however some tools which try and work around this with polling.

So now we have another issue. The develop/test process might be nice on Linux, but for devs on other platforms the process is more clunky.

Docker for Mac and Windows to the rescue

The issues I’ve mentioned so far are the big ones which cause me problems personally, I’m sure there are others (please comment and let me know!).

This is why there was rather a lot of interest in the new Docker Beta - one of the big features is that the Docker Machine is going away. In theory, we can use Docker on a Mac or Windows and have the same experience as on Linux.

So how?

Virtualbox is gone. We still need a VM, but this VM is now a very lightweight Alpine Linux based image which runs on xhyve for MacOS and Hyper-V for Windows. All management of this VM is handled by the docker executable.

If these are not familiar terms, Alpine Linux is an extreeeemely lightweight Linux distro originally design to fit on a floppy disk (I think it clocks at around 5 MB now). xhyve is an extremely lightweight hypervisor which allows FreeBSD and some other distros on OSX. Hyper-V is a native hypervisor for Windows Server which can run on Windows 8 onwards⁵.

Using tools specifically designed for each platform (and with the help of both Apple and Microsoft), Docker have been able to make the experience much more seamless and smooth.

Trying It Out

Removing the three pain points discussed and a clean and simple setup process is what I’m looking at today, and here’s the results.

Installation

Piece of cake. Download the beta, install, run, enter the beta key and pop, there’s the new docker:

The new status bar icon gives me a way to quickly see the status of the machine. Some of the commands hint at features to come, others offer the instructions needed. Settings are fairly basic, but I’m not sure what else you’d need:

1. Localhost vs docker-machine IP

Quickly bashing out the commands below shows that the virtual machine IP address issue is gone:

docker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123 mysql
mysql -uroot -p123 -h127.0.0.1
> show databases;

Great news!

How this works under the hood is a mystery to me. If anyone knows, I’d be interested and would like to update this writeup!

2. Terminal Hassle

Quick and easy to test - running any terminal any way I like lets me access containers using the docker executable - no magic needed:

Here’s a screenshot of iTerm3, the Terminal App and the Terminal App running zsh, all of which are happily communicating with the docker deamon through the docker app.

3. In Container Development

I’ve not thrashed this one too hard, but gone for a quick sanity check. Throwing together probably my best ever node.js app⁶:

main.js

setInterval(function() {
console.log("Goodbye, cruel world!");
}, 1000);

and a simple dockerfile:

Dockerfile

FROM node:6
WORKDIR src/
ADD package.json .
RUN npm install
CMD npm start

is enough to test this. I can build then run the container, mounting the working directory into the src volume on the container:

docker build -t incontainerdev .
docker run -it -v `pwd`:/src incontainerdev]

Immediately, I open a new window and change the source code and save (on my local Mac, not in the container). Voila:

Live reloading works without a hitch! nodemon picks up my changes, using inotify from the VM (all through a lightweight userspace hypervisor).

You know what is cool⁷? I don’t even need Node.js installed to build this Node app! The runtime is in the container, all of the execution happens in the container.

That’s a Wrap

That’s it for my initial impressions. From this point onwards I’m going to be using Docker for Mac heavily as I’ll do all of my work with it installed, so from time to time I may update this article with other observations.

The key takeaway is: at the moment, Docker for Mac just works. I’m using it in the same way I would on Ubuntu with no messing around. This is great, it seems like a simple thing but I’m guessing it was a lot of effort from the guys and girls at Docker, Microsoft and Apple.

This is still a Beta, there’ll be bugs and they’ll be fixed. I can’t wait for the Beta to go fully into the wild, and see what exciting things people can do with it.

As usual, any comments or observations are welcome!

References

Further Reading

Namespaces: http://man7.org/linux/man-pages/man7/namespaces.7.html cgroups: http://man7.org/linux/man-pages/man7/cgroups.7.html Docker Execution Drivers: https://blog.docker.com/2014/03/docker-0-9-introducing-execution-drivers-and-libcontainer/ inotify: http://man7.org/linux/man-pages/man7/inotify.7.html The challenges of in-container development on OSX: http://hharnisc.github.io/2015/09/16/developing-inside-docker-containers-with-osx.html

Read up on namespaces here and cgroups here. Docker can also use LXC but no longer has to, there’s a great write-up here. ↩︎
Surprise! ↩︎
Read up on inotify here ↩︎
The issue will not be resolved: https://www.virtualbox.org/ticket/10660 ↩︎
I’ve not used the Docker for Windows Beta yet so have not got first hand experience of it. I’ve also not looked into compatibility, from memory Hyper-V isn’t available on Home versions of Windows, but I might be wrong. ↩︎
Inspired by my first programming book, the excellent C for Dummies by Dan Gookin. ↩︎
For a given definition of cool. ↩︎

Learn Docker by building a Microservice

Tue, 19 Apr 2016 08:54:39 +0000

If you are looking to get your hands dirty and learn all about Docker, then look no further!

In this article I’m going to show you how Docker works, what all the fuss is about, and how Docker can help with a basic development task - building a microservice.

We’ll use a simple Node.js service with a MySQL backend as an example, going from code running locally to containers running a microservice and database.

Once you’ve read the article, you can find the source code here:

github.com/dwmkerr/node-docker-microservice

What is Docker?

At its heart, Docker is software which lets you create an image (which is a lot like a template for a virtual machine) and then run instances of that image in a container.

Docker maintains a vast repository of images, called the Docker Hub which you can use as starting points or as free storage for your own images. You can install Docker, choose an image you’d like to use, then run an instance of it in a container.

We’re going to build images, create containers from images and more in this article.

Install Docker

To follow along and use this article, you’ll need Docker.

Check the installation guide for your platform, docs.docker.com/engine/installation.

If you are on Mac or Windows, consider using a Virtual Machine. I use Parallels on Mac OS X to run an Ubuntu machine for most development activities. Being able to take snapshots, break things and then revert back is very handy when experimenting.

Try It Out

Enter this command:

docker run -it ubuntu

After a bit of spinning, you’ll see a prompt like this:

root@719059da250d:/#

Try out a few commands and then exit the container:

root@719059da250d:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
root@719059da250d:/# exit

This doesn’t look like much, but a lot has happened!

What you are seeing is the bash shell of an isolated container running Ubuntu, on your machine. It’s yours to place with - you can install things on it, run software, whatever you want.

Here’s a diagram and breakdown of what just happened (the digram is from the ‘Understanding the Architecture’ Docker Documentation, which is great):

We issue a docker command:

docker: run the docker client
run: the command to run a new container
-it: option to give the container an interactive terminal
ubuntu: the image to base the container on

The docker service running on the host (our machine) checks to see if we have a copy of the requested image locally- which there isn’t.
The docker service checks the public registry (the docker hub) to see if there’s an image named ubuntu available- which there is.
The docker service downloads the image and stores it in its local cache of images (ready for next time).
The docker service creates a new container, based on the ubuntu image.

Try any of these:

docker run -it haskell
docker run -it java
docker run -it python

We’re not going to use Haskell today, but you can see, running an environment is very easy.

It’s a snap to build images of our own, with our apps or services on them, databases, whatever we need. We can then run them on any machine with Docker installed - and the image will run in the same, predictable way. We can build our software and the environment it runs on as code and deploy easily.

Let’s look into a simple microservice as an example.

The Brief

We’re going to build a microservice which lets us manage a directory of email addresses to phone numbers, using Node.js and MySQL.

Getting Started

For doing local development we’ll need to install MySQL and create a test database for us to…

…nope.

Creating a local database and running scripts on it is an easy start, but can get messy. Lots of uncontrolled stuff going on. It might work, we could even control it with some shell scripts checked in to our repo, but what if other developers already have MySQL installed? What if they have a database already with the creative name ‘users’ which we want to create?

Step 1: Creating a Test Database Server - in Docker

This is a great Docker use case. We might not want to run our production database in Docker (perhaps we’ll just use Amazon RDS for example), but we can spin up a clean MySQL database in no time as a Docker container for development - leaving our development machine clean and keeping everything we do controlled and repeatable.

Run the following command:

docker run --name db -d -e MYSQL_ROOT_PASSWORD=123 -p 3306:3306 mysql:latest

This starts a MySQL instance running, allowing access through port 3306 using the root password 123.

docker run tells the engine we want to run an image (the image comes at the end, mysql:vlatest
--name db names this container db.
-d (or --detach) detach - i.e. run the container in the background.
-e MYSQL_ROOT_PASSWORD=123 (or --env) environment variables - tells docker we want to provide an environment variable. The variable following it is what the MySQL image checks for setting the default root password.
-p 3306:3306 (or --publish tells the engine that we want to map the port 3306 from inside the container to out port 3306.

The last part is really important - even though that’s the MySQL default port, if we don’t tell docker explicitly we want to map it, it will block access through that port (because containers are isolated until you tell them you want access).

The return value of this function is the container id, a reference to the container which you can use to stop it, restart it, issue commands on it and so on. Let’s see which containers are running:

$ docker ps
CONTAINER ID IMAGE ... NAMES
36e68b966fd0 mysql:latest ... db

The key information is the container ID, image and name. Let’s connect to this image and see what’s there:

$ docker exec -it db /bin/bash
root@36e68b966fd0:/# mysql -uroot -p123
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
+--------------------+
1 rows in set (0.01 sec)
mysql> exit
Bye
root@36e68b966fd0:/# exit

This is pretty clever too:

docker exec -it db tells docker we want to execute a command on the container named db (we could also use the id, or just the first few letters of the id). -it ensures we have an interactive terminal.
mysql -uroot -p123 the command we actually run as a process in the container, which in this case is just the mysql client.

We can create databases, tables, users, whatever we need.

Wrapping up the Test Database

Running MySQL inside a container has already introduced a few Docker tricks, but let’s pause here and move onto the service. For now, we’ll have create a test-database folder with a script to start the database, stop the database and setup test data:

test-database\setup.sql
test-database\start.sh
test-database\stop.sh

Start is simple:

#!/bin/sh

# Run the MySQL container, with a database named 'users' and credentials
# for a users-service user which can access it.
echo "Starting DB..."
docker run --name db -d \
 -e MYSQL_ROOT_PASSWORD=123 \
 -e MYSQL_DATABASE=users -e MYSQL_USER=users_service -e MYSQL_PASSWORD=123 \
 -p 3306:3306 \
 mysql:latest

# Wait for the database service to start up.
echo "Waiting for DB to start up..."
docker exec db mysqladmin --silent --wait=30 -uusers_service -p123 ping || exit 1

# Run the setup script.
echo "Setting up initial data..."
docker exec -i db mysql -uusers_service -p123 users < setup.sql

This script runs the database image in a detached container (i.e. in the background), with a user set up to access a users database, then waits for the database server to start up, then runs a setup.sql script to set initial data.

setup.sql is:


create table directory (user_id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, email TEXT, phone_number TEXT);
insert into directory (email, phone_number) values ('homer@thesimpsons.com', '+1 888 123 1111');
insert into directory (email, phone_number) values ('marge@thesimpsons.com', '+1 888 123 1112');
insert into directory (email, phone_number) values ('maggie@thesimpsons.com', '+1 888 123 1113');
insert into directory (email, phone_number) values ('lisa@thesimpsons.com', '+1 888 123 1114');
insert into directory (email, phone_number) values ('bart@thesimpsons.com', '+1 888 123 1115');

The stop.sh script will stop the container and remove it (containers are left around by docker by default so that they can be restared quickly, we don’t really need that feature for this example):

#!/bin/sh

# Stop the db and remove the container.
docker stop db && docker rm db

We’re going to make this even more slick later on, simplifying this further. Check the code at this stage by looking at the step1 branch of the repo.

Step 2: Creating a Microservice in Node.js

This article is really focused on learning Docker, so I’m not going to spend ages on the Node.js microservice. Instead, I’ll highlight the areas and takeaways.

test-database/ # contains the code seen in Step 1
users-service/ # root of our node.js microservice
- package.json # dependencies, metadata
- index.js # main entrypoint of the app
- api/ # our apis and api tests
- config/ # config for the app
- repository/ # abstraction over our db
- server/ # server setup code

Let’s take this apart bit by bit. The first section to look at is repository. It can be useful to wrap your database access in some kind of class or abstraction, to allow to mock it for testing purposes:

// repository.js
//
// Exposes a single function - 'connect', which returns
// a connected repository. Call 'disconnect' on this object when you're done.
'use strict';

var mysql = require('mysql');

// Class which holds an open connection to a repository
// and exposes some simple functions for accessing data.
class Repository {
 constructor(connection) {
 this.connection = connection;
 }

 getUsers() {
 return new Promise((resolve, reject) => {

 this.connection.query('SELECT email, phone_number FROM directory', (err, results) => {
 if(err) {
 return reject(new Error("An error occured getting the users: " + err));
 }

 resolve((results || []).map((user) => {
 return {
 email: user.email,
 phone_number: user.phone_number
 };
 }));
 });

 });
 }

 getUserByEmail(email) {

 return new Promise((resolve, reject) => {

 // Fetch the customer.
 this.connection.query('SELECT email, phone_number FROM directory WHERE email = ?', [email], (err, results) => {

 if(err) {
 return reject(new Error("An error occured getting the user: " + err));
 }

 if(results.length === 0) {
 resolve(undefined);
 } else {
 resolve({
 email: results[0].email,
 phone_number: results[0].phone_number
 });
 }

 });

 });
 }

 disconnect() {
 this.connection.end();
 }
}

// One and only exported function, returns a connected repo.
module.exports.connect = (connectionSettings) => {
 return new Promise((resolve, reject) => {
 if(!connectionSettings.host) throw new Error("A host must be specified.");
 if(!connectionSettings.user) throw new Error("A user must be specified.");
 if(!connectionSettings.password) throw new Error("A password must be specified.");
 if(!connectionSettings.port) throw new Error("A port must be specified.");

 resolve(new Repository(mysql.createConnection(connectionSettings)));
 });
};

There’s probably a lot of better ways to do this! But basically we can create a Repository object like this:

repository.connect({
 host: "127.0.0.1",
 database: "users",
 user: "users_service",
 password: "123",
 port: 3306
}).then((repo) => {
 repo.getUsers().then(users) => {
 console.log(users);
 });
 repo.getUserByEmail('homer@thesimpsons.com').then((user) => {
 console.log(user);
 })
 // ...when you are done...
 repo.disconnect();
});

There’s also a set of unit tests in the repository/repository.spec.js file. Now that we’ve got a repo, we can create a server. This is server/server.js:

// server.js

var express = require('express');
var morgan = require('morgan');

module.exports.start = (options) => {

 return new Promise((resolve, reject) => {

 // Make sure we have a repository and port provided.
 if(!options.repository) throw new Error("A server must be started with a connected repository.");
 if(!options.port) throw new Error("A server must be started with a port.");

 // Create the app, add some logging.
 var app = express();
 app.use(morgan('dev'));

 // Add the APIs to the app.
 require('../api/users')(app, options);

 // Start the app, creating a running server which we return.
 var server = app.listen(options.port, () => {
 resolve(server);
 });

 });
};

This module exposes a start function, which we can use like this:

var server = require('./server/server);
server.start({port: 8080, repo: repository}).then((svr) => {
 // we've got a running http server :)
});

Notice that server.js uses api/users/js? Here it is:

// users.js
//
// Defines the users api. Add to a server by calling:
// require('./users')
'use strict';

// Only export - adds the API to the app with the given options.
module.exports = (app, options) => {

 app.get('/users', (req, res, next) => {
 options.repository.getUsers().then((users) => {
 res.status(200).send(users.map((user) => { return {
 email: user.email,
 phoneNumber: user.phone_number
 };
 }));
 })
 .catch(next);
 });

 app.get('/search', (req, res) => {

 // Get the email.
 var email = req.query.email;
 if (!email) {
 throw new Error("When searching for a user, the email must be specified, e.g: '/search?email=homer@thesimpsons.com'.");
 }

 // Get the user from the repo.
 options.repository.getUserByEmail(email).then((user) => {

 if(!user) {
 res.status(404).send('User not found.');
 } else {
 res.status(200).send({
 email: user.email,
 phoneNumber: user.phone_number
 });
 }
 })
 .catch(next);

 });
};

Both of these files have unit tests adjacent to the source.

We’ll need config. Rather than using a specialised library, a simple file will do the trick - config/config.js:

// config.js
//
// Simple application configuration. Extend as needed.
module.exports = {
 port: process.env.PORT || 8123,
 db: {
 host: process.env.DATABASE_HOST || '127.0.0.1',
 database: 'users',
 user: 'users_service',
 password: '123',
 port: 3306
 }
};

We can require config as needed. Currently, most config is hard coded, but as you can see from port it’s easy to add environment variables as an option.

Final step - stringing it together with an index.js file which composes everything:

// index.js
//
// Entrypoint to the application. Opens a repository to the MySQL
// server and starts the server.
var server = require('./server/server');
var repository = require('./repository/repository');
var config = require('./config/config');

// Lots of verbose logging when we're starting up...
console.log("--- Customer Service---");
console.log("Connecting to customer repository...");

// Log unhandled exceptions.
process.on('uncaughtException', function(err) {
 console.error('Unhandled Exception', err);
});
process.on('unhandledRejection', function(err, promise){
 console.error('Unhandled Rejection', err);
});

repository.connect({
 host: config.db.host,
 database: config.db.database,
 user: config.db.user,
 password: config.db.password,
 port: config.db.port
}).then((repo) => {
 console.log("Connected. Starting server...");

 return server.start({
 port: config.port,
 repository: repo
 });

}).then((app) => {
 console.log("Server started successfully, running on port " + config.port + ".");
 app.on('close', () => {
 repository.disconnect();
 });
});

We have a little error handling and beyond that we’re just loading config, creating a repo and starting our server.

That’s the microservice. It allows us to get all users, or search a user:

HTTP GET /users # gets all users
HTTP GET /search?email=homer@thesimpons.com # searches by email

If you checkout the code, you’ll see that there’s a few commands available for you:

cd ./users-service
npm install # setup everything
npm test # unit test - no need for a test database running
npm start # run the server - you must have a test database running
npm run debug # run the server in debug mode, opens a browser with the inspector
npm run lint # check to see if the code is beautiful

Asides from the code you’ve seen we have:

Node Inspector for debugging
Mocha/shoud/supertest for unit tests
ESLint for linting

That’s it!

Run the test database with:

cd test-database/
./start.sh

Then the service with:

cd ../users-service/
npm start

You can point your browser to localhost:8123/users and see it in action. If you are using Docker Machine (i.e. you’re on Mac or Windows) then localhost won’t work, you need the IP of the docker machine instead. You can use docker-machine ip to get it.

We’ve whipped through building the service quickly. If you’d like to see this code before we continue, check the step2 branch.

Step 3: Dockerising our Microservice

OK now it gets fun!

So we have a microservice which we can run on a dev box, as long as it has a compatible version of Node.js installed. What we’d like to do is set up our service so that we can create a Docker Image from it, allowing us to deploy our service anywhere which supports docker.

The way we do this is create a Dockerfile. A Dockerfile is a recipe that tells the Docker engine how to build your image. We’ll create a simple Dockerfile in our users-service directory and start to explore how we can adapt it to our needs.

Creating the Dockerfile

Create a new text file called Dockerfile at users-service/ with the content below:

# Use Node v4 as the base image.
FROM node:4
# Run node
CMD ["node"]

Now run the commands below to build the image and run the a container from it:

docker build -t node4 . # Builds a new image
docker run -it node4 # Run a container with this image, interactive

Let’s look at the build command first.

docker build tell the engine we want to create a new image.
-t node4 tag this image with the tag node4. We can refer to this image by tag from now on.
. use the current directory to find the Dockerfile.

After some console output you’ll see we have a new image created. You can see all images on your system with docker images. The next command should be fairly familiar from what we’ve done so far:

docker run run a new container from an image.
-it use an interactive terminal.
node4 the tag of the image we want to use in the container.

When we run this image, we get a node repl, check the current version like so:

> process.version
'v4.4.0'
> process.exit(0)

This is potentially different to the node version on your current machine.

Examining the Dockerfile

Looking at the dockerfile we can see quite easily what is going on:

FROM node:4 the first thing we specify in a Dockerfile is the base image. A quick google finds the node organisation page on the docker hub showing all of the available images. This is essentially bare bones ubuntu with node installed.
CMD ["node"] the CMD command tells docker that this image should run the node executable. When the executable terminates, the container shuts down.

With the addition of a few more commands, we can update our Dockerfile so that it runs our service:

# Use Node v4 as the base image.
FROM node:4
# Add everything in the current directory to our image, in the 'app' folder.
ADD . /app
# Install dependencies
RUN cd /app; \
npm install --production
# Expose our server port.
EXPOSE 8123
# Run our app.
CMD ["node", "/app/index.js"]

The only addition here is that we use the ADD command to copy everything¹ in the current directory to a folder in the container called app/ . We then use RUN to run a command in the image, which installs our modules. Finally, we EXPOSE the server port, telling docker we intend to support inbound connections on 8123, then run our server code.

Ensure the test-database service is running, then build and run the image again:

docker build -t users-service .
docker run -it -p 8123:8123 users-service

If you navigate to localhost:8123/users in a browser you should see an error, checking the console shows our container is reporting some issues:

--- Customer Service---
Connecting to customer repository...
Connected. Starting server...
Server started successfully, running on port 8123.
GET /users 500 23.958 ms - 582
Error: An error occured getting the users: Error: connect ECONNREFUSED 127.0.0.1:3306
at Query._callback (/app/repository/repository.js:21:25)
at Query.Sequence.end (/app/node_modules/mysql/lib/protocol/sequences/Sequence.js:96:24)
at /app/node_modules/mysql/lib/protocol/Protocol.js:399:18
at Array.forEach (native)
at /app/node_modules/mysql/lib/protocol/Protocol.js:398:13
at nextTickCallbackWith0Args (node.js:420:9)
at process._tickCallback (node.js:349:13)

Yikes! So the connection from our users-service container to the test-database container is being refused. We might try running docker ps to see all containers running:

CONTAINER ID IMAGE PORTS NAMES
a97958850c66 users-service 0.0.0.0:8123->8123/tcp kickass_perlman
47f91343db01 mysql:latest 0.0.0.0:3306->3306/tcp db

They’re both there, so what is going on?

Linking Containers

The issue we’ve seen is actually to be expected. Docker containers are supposed to be isolated, so it wouldn’t make much sense if we could create connections between containers without us explicitly allowing it.

Yes, we can connect from our machine (the host) to a container, because we’ve opened ports for that (using the -p 8123:8123 argument for example). If we allowed containers to talk to each other in the same way, then two containers running on the same machine would be able to communicate, even if the developers didn’t intend it, and that’s a recipe for disaster, especially when we might have a cluster of machines whos job it is to run containers from different applications.

If we’re going to connect from one container to another, we need to link them, which tells docker that we explicitly want to allow communication between the two. There are two ways of doing this, the first is the ‘old fasioned’ but quite simple way, the second we’ll see a little later.

Linking Containers with the ’link’ parameter

When we run a container, we can tell docker that we intend to connect to another container using the link parameter. In our case, we can run our service correctly like this:

docker run -it -p 8123:8123 --link db:db -e DATABASE_HOST=DB users-service

docker run -it run a docker image in a container, with an interactive terminal.
-p 8123:8123 map the host port 8123 to the container port 8123.
link db:db link to the container named db and refer to it as db.
-e DATABASE_HOST=db set the DATABASE_HOST environment variable to db.
users-service the name of the image to run in our container.

Now when we go to localhost:8123/users everything works.

How it works

Remember our config file for the service? It allowed us to specify a database host with an environment variable:

// config.js
//
// Simple application configuration. Extend as needed.
module.exports = {
 port: process.env.PORT || 8123,
 db: {
 host: process.env.DATABASE_HOST || '127.0.0.1',
 database: 'users',
 user: 'users_service',
 password: '123',
 port: 3306
 }
};

When we run the container, we set this environment variable to DB, which means we’re connecting to a host called DB. This is automatically set up for us by the docker engine when we link to a container.

To see this in action, try running docker ps to list all running containers. Look up the name of the container running the users-service, which will be a random name such as trusting_jang:

docker ps
CONTAINER ID IMAGE ... NAMES
ac9449d3d552 users-service ... trusting_jang
47f91343db01 mysql:latest ... db

Now we can look at the hosts available on our container:

docker exec trusting_jang cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2 db 47f91343db01 # linking magic!!
172.17.0.3 ac9449d3d552

Remember how docker exec works? Choose a container name and then whatever follows is the command you’ll execute on the container, in our case cat /etc/hosts.

OK the hosts file doesn’t have the # linking magic!! comment, that’s so you can see - docker has added db to our hosts file so we can refer to the linked container by hostname. This is one consequence of linking. Here’s the other:

docker exec trusting_jang printenv | grep DB
DB_PORT=tcp://172.17.0.2:3306
DB_PORT_3306_TCP=tcp://172.17.0.2:3306
DB_PORT_3306_TCP_ADDR=172.17.0.2
DB_PORT_3306_TCP_PORT=3306
DB_PORT_3306_TCP_PROTO=tcp
DB_NAME=/trusting_jang/db

From this command we can also see that when docker links a container, it also provides a set of environment variables with some helpful information. We know the host, tcp port and container name.

That’s step 3 complete - we have a MySQL database running happily in a container, we have a node.js microservice which we can run locally or in a container of its own, and we know how to link them together.

You can check out how the code looks at this stage by going to the step3 branch.

Step 4: Integration Testing the Environment

We can now write an integration test which calls the actual server, running as a docker container, calling the containerised test database.

Writing the integration test can be done in whatever language or on whatever platform you want, within reason, but to keep things simple I’m using Node.js as we’ve already seen Mocha and Supertest in our project.

In a new folder, called integration-tests we’ve got a single index.js:

var supertest = require('supertest');
var should = require('should');

describe('users-service', () => {

 var api = supertest('http://localhost:8123');

 it('returns a 200 for a known user', (done) => {

 api.get('/search?email=homer@thesimpsons.com')
 .expect(200, done);
 });

});

This will check an API call and show the results of the test².

As long as your users-service and test-database are running, the tests will pass. However, at this stage the services are getting a little harder to handle:

We have to use a shell script to start and stop the database
We have to remember a sequence of commands to start the users service against the database
We have to use node directly to run the integration tests

Now that we’re a little more familiar with Docker we can fix these issues.

Simplifiying the Test Database

Currently we have the following files for the test database:

/test-database/start.sh
/test-database/stop.sh
/test-database/setup.sql

Now that we’re more familar with Docker, we can improve on this. Looking into the mysql image documentation on Docker Hub there’s a note which tells us any .sql or .sh file added to the image’s /docker-entrypoint-initdb.d folder will be executed when setting up the DB.

This means we can replace our start.sh and stop.sh scripts with a Dockerfile:

FROM mysql:5
ENV MYSQL_ROOT_PASSWORD 123
ENV MYSQL_DATABASE users
ENV MYSQL_USER users_service
ENV MYSQL_PASSWORD 123
ADD setup.sql /docker-entrypoint-initdb.d

Now to run our test database it is just:

docker build -t test-database .
docker run --name db test-database

Composing

Building and running each container is still somewhat time consuming. We can take things a step further with the Docker Compose tool.

Docker Compose lets you create a file which defines each container in your system, the relationships between them, and build or run them all.

First, install Docker Compose. Now create a new file in the root of your project called docker-compose.yml:

version: '2'
services:
users-service:
build: ./users-service
ports:
- "8123:8123"
depends_on:
- db
environment:
- DATABASE_HOST=db
db:
build: ./test-database

Now check this out:

docker-compose build
docker-compose up

Docker Compose has built all of the images needed for our application, created containers fromthem, run them in the correct order and started the whole stack!

The docker-compose build command builds each image which is listed in the docker-compose.yml file:

version: '2'
services:
users-service:
build: ./users-service
ports:
- "8123:8123"
depends_on:
- db
environment:
- DATABASE_HOST=db
db:
build: ./test-database

The build value for each of our services tells docker where to go to find the Dockerfile. When we run docker-compose up, docker starts all of our services. Notice from the Dockerfile we can specify ports and dependencies. Actually, there’s a whole bunch of config we can change here.

In another terminal, run docker compose down to gracefully shut down the containers.

Winding Up

We’ve seen a lot of docker in this article, but there’s a lot more to it. I hope this has shown some of the interesting and useful things that you can use docker for in your workflow.

As usual, questions and comments are welcomed! I’d also strongly recommend the document Understanding Docker to get a deeper understanding of how docker works.

You can see the final source code for the project built in this article at github.com/dwmkerr/node-docker-microservice

Notes

Copying everything is actually a bad idea, because we will also copy the node_modules folder. Generally it is a better idea explicitly list the files or folders you want to copy, or use a .dockerignore file, which works just like the .gitignore file. ↩︎
If the server isn’t running, it will actually show a rather annoying exception, due to a bug in supertest, see github.com/visionmedia/supertest/issues/314. ↩︎