Kubernetes on dwmkerr.com

Observations, tips and tricks for the CKA certification

Thu, 04 Jun 2020 00:00:00 +0000

In this article I’ll share some observations, tips and tricks for the Linux Foundation’s “Certified Kubernetes Administrator certification and exam.

I’ve been operating Kubernetes in multiple environments for a few years now. I thought this would be an easy certification to get, but I was surprised by how hard it was!

I took this exam without doing any formal training, I mostly focused on the areas of the curriculum which I knew I was a little weak at. The task-based structure for the exam I thought was really excellent. It took me two attempts to pass, and I learnt a few things along the way.

Here I’ll share some thoughts on the certification which hopefully will be useful if you are considering taking it!

Tip: Do the right Certification!
Tip: Understand the Format!
Tip: Know your Vim
You need to know the architecture of Kubernetes
Tip: You Need to know Linux Sysadmin
Tip: “Dry Run” is your friend
Tip: Know how to troubleshoot networking
Tip: Nail the easy questions quickly
That’s it!

Tip: Do the right Certification!

The CKA exam tests administration and operation skills and techniques for Kubernetes. If you have set up and administered clusters before, this will likely not be too challenging. But if you’ve never set up a cluster by hand, troubleshot weird issues, fixed clusters and so on, then this is likely going to be very hard.

There is a certification which is much more geared towards developers who use Kubernetes, but don’t necessarily administer it - that’s the CKAD exam and might be the one to take if you are not too familiar with system administration.

Tip: Understand the Format!

This is not a multiple choice question exam. It’s a task based exam, meaning you have about 22 or so specific tasks to complete, in a web browser which has a terminal connected to a cluster.

It is open-book - meaning that you can use the Kubernetes Documentation during the exam. It’s not a memory test of specific flags for commands or whatever, it will really require you to work with a running cluster. This means you’ll have to be pretty familiar with kubectl, kubeadm and also Linux in general!

Tip: Know your Vim

In the two exams I took, nano was available. But if you are using nano to work with files you may struggle for time.

I spent a lot of time in vim in the exam. vim is my main text editor for day to day work, so I’m fairly familiar with it. Knowing how to quickly copy a file (lets say for example a file which represents a deployment) and quickly manipulate the text in it will be crucial. Make sure you are going to be using a text editor which you can be efficient in!

You won’t be using a graphical text editor to work with files, so being competent in a terminal editor like vim or emacs could make a big difference. Of course you could install your favourite text editor, but you won’t be able to use a graphical editor like VS Code.

Also, as in most Linux distributions, screen is available out of the box, and tmux can also be installed. If you are familiar with either of these terminal mutliplexers it could save you a tonne of time, for example being able to run watch -n 5 -d kubectl get pods in one pane while applying resources in another.

You need to know the architecture of Kubernetes

This exam will require you to deal with trivial tasks such as running a deployment or creating a volume. But the questions which focus on that tend to only count for one or two percent of the overall grade each. Questions which deal with troubleshooting actual Kubernetes issues could count for six or seven percent each.

This means you need to know how Kubernetes is architecture. The kubelet which runs on nodes, the API server, the etcd store, all of these things you have to understand how they work and how they fit together.

The online documentation covers the architecture in detail, here’s the best place to start:

https://kubernetes.io/docs/concepts/overview/components/

You will need to know how the control plane works, how nodes communicate, how transport of messages works and is secured if you are going to have a chance at dealing with the harder questions.

Tip: You Need to know Linux Sysadmin

If you are not familiar with systemctl, journalctl, apt, systemd units and how the core Kubernetes components are configured, you’ll really struggle.

Look over the CNCF curriculum - expect to not just have to know how to deal with ‘happy path’ situations, but also broken clusters, incorrect configuration and so on.

Tip: “Dry Run” is your friend

One thing which helped me a lot in my second attempt at the exam was the --dry-run flag. Before you create resources or change anything, run the operation with the --dry-run flag and see whether the output is what you would expect.

This is a quick and easy way to see the changes to the cluster which you are going to apply - and troubleshoot them - before making any actual changes.

Tip: Know how to troubleshoot networking

Networking in Kubernetes is complex. You must be able to troubleshoot networking issues in the cluster to be able to deal with the more complex tasks.

This means that you should know how to be able to run typical networking tools like dig, nslookup, telnet etc, in the cluster itself.

If you are not familiar with these tools you might need to take an online course in Kubernetes or Linux Networking Administration before considering this certification. The Linux Certified Systems Administrator training would be a good place to start.

If you have taken the Docker Certified Associate exam then some of this should be familiar. If you are not very familiar with how Docker itself works, you’ll likely struggle with Kubernetes.

Tip: Nail the easy questions quickly

There are a lot of tasks which only count for one or two percent each; these ones you should be able to complete in a few minutes. You’ll need all the time in the exam to work on the really hard questions which deal with diagnosing and fixing cluster issues.

Know your core Kubernetes concepts; if you have done the CKAD exam you should be good, if not, check the curriculum and make sure you can quickly complete all of the trivial tasks without wasting too much time.

That’s it!

Hopefully this was helpful! Good luck if you are taking the exam and hopefully you’ll find it a challenging but rewarding experience. I’ve taken many exams over the years but this was one of the most challenging, but also one of the most enjoyable, I really felt like it was testing practical techniques rather than your ability to just remember random commands and flags.

As always, if you have any comments or questions, please just add them in the section below!

Manipulating Istio and other Custom Kubernetes Resources in Golang

Mon, 08 Oct 2018 21:34:02 +0000

In this article I’ll demonstrate how to use Golang to manipulate Kubernetes Custom Resources, with Istio as an example. No knowledge of Istio is needed, I’ll just use it to demonstrate the concepts!

Istio is a highly popular Service Mesh platform which allows engineers to quickly add telemetry, advanced traffic management and more to their service-based applications.

One interesting element of how Istio works is that when deployed into a Kubernetes cluster, many key configuration objects are handled as Custom Resources. Custom Resources are a very powerful Kubernetes feature, which allow you to create your own ‘first class’ resources (just like Pods, ReplicaSets, Deployments or whatever) and then interface with them using kubectl or the Kubernetes APIs.

In this article I’ll show you how to interface with these Custom Resources using the Golang Kubernetes client.

CRDs: A Quick Overview

When you set up Istio for your cluster, one common thing you will likely do is specify how you will route traffic. This can be quite sophisticated, as shown below:

Figure 1: Istio Traffic Management Examples, from istio.io

One way for a system like this to be configured would be to have a ConfigMap which contains the definition of how services are routed.

However, Istio actually registers new types of resources (Custom Resource Definitions) which represent things like Gateways or Services. We can create/update/delete/manipulate them just like any other Kubernetes object.

For example, I could create a virtual service for the example above with something like this:

cat << EOF | kubectl create -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: service2
spec:
 hosts:
 - "*"
 gateways:
 - demo1-gateway
 http:
 - route:
 - destination:
 host: service2
 subset: v1
 weight: 95
 - destination:
 host: service2
 subset: v2
 weight: 5
EOF

Again, the important thing is not the specific content of this resource, more the fact that I can treat my Istio resources just like I would any other Kubernetes object:

$ kubectl get virtualservices.networking.istio.io
NAME AGE
service2 93s

Or:

$ kubectl delete virtualservices.networking.istio.io/service2

I can use edit, describe, register lifecycle events, watch for changes, and so on.

Working with CRDs in Golang

The Golang Kubernetes Client allows you to create strongly defined types which you can then use to interface with CRDs. An example is in the Red Hat blog post Kubernetes Deep Dive: Code Generation for Custom Resources.

This is an excellent approach, but can feel pretty heavy if you want to quickly access some data, and don’t want to have to generate a lot of code.

There is an alternative, which is to use the DynamicClient. The preferred approach seems to be the first, which involves code generation, so little documentation exists for the second approach. However, it is actually very simple.

Here’s an example of how you can list all Istio VirtualService resources, without having to generate any code:

import (
 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/dynamic"
)

// Create a Dynamic Client to interface with CRDs.
dynamicClient, _ := dynamic.NewForConfig(config)

// Create a GVR which represents an Istio Virtual Service.
virtualServiceGVR := schema.GroupVersionResource{
 Group: "networking.istio.io",
 Version: "v1alpha3",
 Resource: "virtualservices",
}

// List all of the Virtual Services.
virtualServices, _ := dynamicClient.Resource(virtualServiceGVR).Namespace("default").List(metav1.ListOptions{})
for _, virtualService := range virtualServices.Items {
 fmt.Printf("VirtualService: %s\n", virtualService.GetName())
}

This snippet omits setup and error-handling for clarity, the full example is in the k8s-list-virtualservices.go gist.

Patching CRDs in Golang

You may have noticed that the .Resource().Namespace().List() code looks very similar to the structure for making API calls when using the Kubernetes Clientset. In fact, it is essentially the same. Looking at the interface, you can see you have all of the operations you’d expect:

Create
Update
Delete
Get

And so on. This is nice because you can use the same trick in my article ‘Patching Kubernetes Resources in Golang’ to manipulate these entities, without ever having to create a structure to represent it.

Here’s another abbreviated example, this time showing how we can adjust the weight of the routing from the services to 50%/50%:

import (
 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/dynamic"
)

// Create a GVR which represents an Istio Virtual Service.
virtualServiceGVR := schema.GroupVersionResource{
 Group: "networking.istio.io",
 Version: "v1alpha3",
 Resource: "virtualservices",
}

// Weight the two routes - 50/50.
patchPayload := make([]PatchUInt32Value, 2)
patchPayload[0].Op = "replace"
patchPayload[0].Path = "/spec/http/0/route/0/weight"
patchPayload[0].Value = 50
patchPayload[1].Op = "replace"
patchPayload[1].Path = "/spec/http/0/route/1/weight"
patchPayload[1].Value = 50
patchBytes, _ := json.Marshal(patchPayload)

// Apply the patch to the 'service2' service.
_, err := dynamicClient.Resource(virtualServiceGVR).Namespace("default").Patch("service2", types.JSONPatchType, patchBytes)

See the full example in the gist k8s-patch-virtualservice.go

After running the sample, you can use the Kubernetes CLI to verify the changes:

$ kubectl get virtualservices.networking.istio.io/service2 -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
clusterName: ""
creationTimestamp: 2018-10-08T09:53:16Z
generation: 0
name: service2
namespace: default
resourceVersion: "487435"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/service2
uid: fac5930c-cadf-11e8-90a2-42010a94005b
spec:
gateways:
- demo1-gateway
hosts:
- '*'
http:
- route:
- destination:
host: service2
subset: v1
weight: 50
- destination:
host: service2
subset: v2
weight: 50

Keep It Simple!

That’s it! This trick made something I was working on a lot easier, but it took a little bit of experimentation to get right. I hope you find the approach useful. Please share any thoughts/questions in the comments.

Patching Kubernetes Resources in Golang

Tue, 24 Jul 2018 06:33:17 +0000

Recently I needed to be able to quickly adjust the number of replicas in a Kubernetes Replication Controller. The original solution I’d seen pulled down the spec, modified it, then updated it. There’s a better way!

There’s a patch API for Kubernetes resources. Patching resources is faster and easier than pulling them and updating the spec wholesale. However, the documentation is a little limited.

After some trial and error I got it working, here’s the solution. I thought it might be helpful to share for others!

The Solution

I’ll start with the solution. If this is all you need, you are good to go. The details of how this works are presented afterwards. In this example I’ll update the number of replicas in the my-rc controller:

package main

import (
 "encoding/json"
 "fmt"

 types "k8s.io/apimachinery/pkg/types"
 "k8s.io/client-go/kubernetes"
 _ "k8s.io/client-go/plugin/pkg/client/auth"
 "k8s.io/client-go/tools/clientcmd"
)

var (
 // Leave blank for the default context in your kube config.
 context = ""

 // Name of the replication controller to scale, and the desired number of replicas.
 replicationControllerName = "my-rc"
 replicas = uint32(3)
)

// patchStringValue specifies a patch operation for a string.
type patchStringValue struct {
 Op string `json:"op"`
 Path string `json:"path"`
 Value string `json:"value"`
}

// patchStringValue specifies a patch operation for a uint32.
type patchUInt32Value struct {
 Op string `json:"op"`
 Path string `json:"path"`
 Value uint32 `json:"value"`
}

func scaleReplicationController(clientSet *kubernetes.Clientset, replicasetName string, scale uint32) error {
 payload := []patchUInt32Value{{
 Op: "replace",
 Path: "/spec/replicas",
 Value: scale,
 }}
 payloadBytes, _ := json.Marshal(payload)
 _, err := clientSet.
 CoreV1().
 ReplicationControllers("default").
 Patch(replicasetName, types.JSONPatchType, payloadBytes)
 return err
}

func main() {
 // Get the local kube config.
 fmt.Printf("Connecting to Kubernetes Context %v\n", context)
 config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
 clientcmd.NewDefaultClientConfigLoadingRules(),
 &clientcmd.ConfigOverrides{CurrentContext: context}).ClientConfig()
 if err != nil {
 panic(err.Error())
 }

 // Creates the clientset
 clientset, err := kubernetes.NewForConfig(config)
 if err != nil {
 panic(err.Error())
 }

 // Scale our replication controller.
 fmt.Printf("Scaling replication controller %v to %v\n", replicationControllerName, replicas)
 err = scaleReplicationController(clientset, replicationControllerName, replicas)
 if err != nil {
 panic(err.Error())
 }
}

This code is also available in the k8s-patch.go gist.

The Mechanism

The Kubernetes Patch API supports a few different methods for modifying resources. It is important to be aware that there is not a universally accepted ‘standard’ approach to representing a change to a resource in a REST API.

There are three strategies you can use to patch:

merge: follows the JSON Merge Patch Spec (RFC 7386)
stragetic: A strategic merge, which addresses some limitations of the merge patch (noted in this doc.
json: follows the JSON Patch Spec (RFC 6902)

These are documented in detail at:

docs/devel/api-conventions.md#patch-operations

The mechanism I’ve used here is json, which I think is the clearest to the reader. To use this strategy we need to build a payload describing what we are changing. This might look like this:

{
 "op": "replace",
 "path": "/spec/replicas",
 "value": 4
}

The op field can be remove, replace, add etc etc (all the details are in the RFC 6902), or the slightly more readable jsonpatch.com). This allows the operation to be very explicit to the reader, which is helpful. We create a struct which represents an operation on a string or integer (or whatever data type we need), serialize it and pass to the API.

Under the hood, the Golang client will simply translate this into an HTTP call which will look like something like this:

PATCH /api/v1/namespaces/default/replicationcontrollers/app-server-blue HTTP/1.1
Host: 127.0.0.1
Content-Type: application/json-patch+json
Content-Length: 70
[{
"op": "replace",
"path": "/spec/replicas",
"value": 4
}]

This corresponds to the documentation on the Patch Operations. Note that the patch operation type is specified in the Content-Type header.

Hopefully this’ll help you if you need to patch resources, are struggling with the docs and are a Go noob like me! Any tips on how to make the code cleaner or more idomatic would be welcome.

Thanks to the following articles and issues which helped me unpick this:

The Death of Microservice Madness in 2018

Fri, 12 Jan 2018 10:52:25 +0000

En Español | Reddit Thread | Hacker News Thread

Microservices became a very popular topic over the last couple of years¹. ‘Microservice madness’ goes something like this:

Netflix are great at devops. Netflix do microservices. Therefore: If I do microservices, I am great at devops.

There are many cases where great efforts have been made to adopt microservice patterns without necessarily understanding how the costs and benefits will apply to the specifics of the problem at hand.

I’m going to describe in detail what microservices are, why the pattern is so appealing, and also some of the key challenges that they present.

I’ll finish with a set of simple questions might be valuable to ask yourself when you are considering whether microservices are the right pattern for you. The questions are at the end of the article.

What are microservices, and why are they so popular?

Let’s start with the basics. Here is how a hypothetical video sharing platform might be implemented, first in the form of a monolith (single large unit) and then in the form of microservices:

The difference between the two systems is that the first is a single large unit; a monolith. The second is a set of small, specific services. Each service has a specific role.

When the diagram is drawn at this level of detail, it is easy to see the appeal. There are a whole host of potential benefits:

Independent Development: Small, independent components can be built by small, independent teams. A group can work on a change to the ‘Upload’ service without interfering with the ‘Transcode’ service, or even knowing about it. The amount of time to learn about a component is greatly reduced, and it is easier to develop new features.

Independent Deployment: Each individual component can be deployed independently. This allows new features to be released with greater velocity and less risk. Fixes or features for the ‘Streaming’ component can be deployed without requiring other components to be deployed.

Independent Scalability: Each component can be scaled independently of each other. During busy periods when new shows are released, the ‘Download’ component can be scaled up to handle the increased load, without having to scale up every component, which makes elastic scaling more feasible and reduces costs.

Reusability: Components fulfil a small, specific function. This means that they can more easily be adapted for use in other systems, services or products. The ‘Transcode’ component could be used by other business units, or even turned into a new business, perhaps offering transcoding services for other groups.

At this level of detail, the benefits of a microservice model over a monolithic model seem obvious. So if that’s the case - why is this pattern only recently in vogue? Where has it been all my life?

If this is so great, why hasn’t it been done before?

There are two answers to this question. One is that it has - to the best of our technical capabilities, and the other is that more recent technical advances have allowed us to take it to a new level.

When I started writing the answer to this question, it turned into a long description, so I’m actually going to separate it into another article and publish it a little later². At this stage, I will skip the journey from single program to many programs, ignore ESBs and Service Orientated Architecture, component design and bounded contexts, and so on.

Those who are interested can read more about the journey separately. Instead I’ll say that in many ways we’ve been doing this for a while, but with the recent explosion in popularity of container technology (Docker in particular) and in orchestration technology (such as Kubernetes, Mesos, Consul and so on) this pattern has become much more viable to implement from a technical standpoint.

So if we take it as a given that we can implement a microservice arrangement, we need to think carefully about the should. We’ve seen the high-level theoretical benefits, but what about the challenges?

What’s the problem with microservices?

If microservices are so great, what’s the big deal? Here are some of the biggest issues I’ve seen.

Increased complexity for developers

Things can get a lot harder for developers. In the case where a developer wants to work on a journey, or feature which might span many services, that developer has to run them all on their machine, or connect to them. This is often more complex than simply running a single program.

This challenge can be partially mitigated with tooling³, but as the number of services which makes up a system increases, the more challenges developers will face when running the system as a whole.

Increased complexity for operators

For teams who don’t develop services, but maintain them, there is an explosion in potential complexity. Instead of perhaps managing a few running services, they are managing dozens, hundreds or thousands of running services. There are more services, more communication paths, and more areas of potential failure.

Increased complexity for devops

Reading the two points above, it may grate that operations and development are treated separately, especially given the popularity of devops as a practice (which I am a big proponent of). Doesn’t devops mitigate this?

The challenge is that many organisations still run with separated development and operations teams - and a organisation that does is much more likely to struggle with adoption of microservices.

For organisations which have adopted devops, it’s still hard. Being both a developer and an operator is already tough (but critical to build good software), but having to also understand the nuances of container orchestration systems, particularly systems which are evolving at a rapid pace, is very hard. Which brings me onto the next point.

It requires serious expertise

When done by experts, the results can be wonderful. But imagine an organisation where perhaps things are not running smoothly with a single monolithic system. What possible reason would there be that things would be any better by increasing the number of systems, which increases the operational complexity?

Yes, with effective automation, monitoring, orchestration and so on, this is all possible. But the challenge is rarely the technology - the challenge is finding people who can use it effectively. These skillsets are currently in very high demand, and may be difficult to find.

Real world systems often have poorly defined boundaries

In all of the examples we used to describe the benefits of microservices, we spoke about independent components. However in many cases components are simply not independent. On paper, certain domains may look bounded, but as you get into the muddy details, you may find that they are more challenging to model than you anticipated.

This is where things can get extremely complex. If your boundaries are actually not well defined, then what happens is that even though theoretically services can be deployed in isolation, you find that due to the inter-dependencies between services, you have to deploy sets of services as a group.

This then means that you need to manage coherent versions of services which are proven and tested when working together, you don’t actually have an independently deployable system, because to deploy a new feature, you need to carefully orchestrate the simultaneous deployment of many services.

The complexities of state are often ignored

In the previous example, I mentioned that a feature deployment may require the simultaneous rollout of many versions of many services in tandem. It is tempting to assume that sensible deployment techniques will mitigate this, for example blue/green deployments (which most service orchestration platforms handle with little effort), or multiple versions of a service being run in parallel, with consuming channels deciding which version to use.

These techniques mitigate a large number of the challenges if the services are stateless. But stateless services are quite frankly, easy to deal with. In fact, if you have stateless services, then I’d be inclined to consider skipping microservices altogether and consider using a serverless model.

In reality, many services require state. An example from our video sharing platform might be the subscription service. A new version of the subscriptions service may store data in the subscriptions database in a different shape. If you are running both services in parallel, you are running the system with two schemas at once. If you do a blue green deployment, and other services depend on data in the new shape, then they must be updated at the same time, and if the subscription service deployment fails and rolls back, they might need to roll back too, with cascading consequences.

Again, it might be tempting to think that with NoSQL databases these issues of schema go away, but they don’t. Databases which don’t enforce schema do not lead to schemaless systems - they just mean that schema tends to be managed at the application level, rather than the database level. The fundamental challenge of understanding the shape of your data, and how it evolves, cannot be eliminated.

The complexitities of communication are often ignored

As you build a large network of services which depend on each other, the liklihood is that there will be a lot of inter-service communication. This leads to a few challenges. Firstly, there are a lot more points at which things can fail. We must expect that network calls will fail, which means when one service calls another, it should expect to have to retry a number of times at the least. Now when a service has to potentially call many services, we end up in a complicated situation.

Imagine a user uploads a video in the video sharing service. We might need to run the upload service, pass data to the transcode service, update subscriptions, update recommendations and so on. All of these calls require a degree of orchestration, if things fail we need to retry.

This retry logic can get hard to manage. Trying to do things synchronously often ends up being untenable, there are too many points of failure. In this case, a more reliable solution is to use asynchronous patterns to handle communication. The challenge here is that asynchronous patterns inherently make a system stateful. As mentioned in the previous point, stateful systems and systems with distributed state are very hard to handle.

When a microservice system uses message queues for intra-service communication, you essentially have a large database (the message queue or broker) glueing the services together. Again, although it might not seem like a challenge at first, schema will come back to bite you. A service at version X might write a message with a certain format, services which depend on this message will also need to be updated when the sending service changes the details of the message it sends.

It is possible to have services which can handle messages in many different formats, but this is hard to manage. Now when deploying new versions of services, you will have times where two different versions of a service may be trying to process messages from the same queue, perhaps even messages sent by different versions of a sending service. This can lead to complicated edge cases. To avoid these edge cases, it may be easier to only allow certain versions of messages to exist, meaning that you need to deploy a set of versions of a set of services as a coherent whole, ensuring messages of older versions are drained appropriately first.

This highlights again that the idea of independent deployments may not hold as expected when you get into the details.

Versioning can be hard

To mitigate the challenges mentioned previously, versioning needs to be very carefully managed. Again, there can be a tendency to assume that following a standard such as semver[4] will solve the problem. It doesn’t. Semver is a sensible convention to use, but you will still have to track the versions of services and APIs which can work together.

This can get very challenging very quickly, and may get to the point where you don’t know which versions of services will actually work properly together.

Managing dependencies in software systems is notoriously hard, whether it is node modules, Java modules, C libraries or whatever. The challenges of conflicts between independent components when consumed by a single entity are very hard to deal with.

These challenges are hard to deal with when the dependencies are static, and can be patched, updated, edited and so on, but if the dependencies are themselves live services, then you may not be able to just update them - you may have to run many versions (with the challenges already described) or bring down the system until it is fixed holistically.

Distributed Transactions

In situations where you need transaction integrity across an operation, microservices can be very painful. Distributed state is hard to deal with, many small units which can fail make orchestrating transactions very hard.

It may be tempting to attempt to avoid the problem by making operations idempotent, offering retry mechanisms and so on, and in many cases this might work. But you may have scenarios where you simply need a transaction to fail or succeed, and never be in an intermediate state. The effort involved in working around this or implementing it in a microservice model may be very high.

Microservices can be monoliths in disguise

Yes, individual services and components may be deployed in isolation, however in most cases you are going to have to be running some kind of orchestration platform, such as Kubernetes. If you are using a managed service, such as Google’s GKE⁴ or Amazon’s EKS⁵, then a large amount of the complexity of managing the cluster is handled for you.

However, if you are managing the cluster yourself, you are managing a large, complicated, mission critical system. Although the individual services may have all of the benefits described earlier, you need to very carefully manage your cluster. Deployments of this system can be hard, updates can be hard, failover can be hard and so on.

In many cases the overall benefits are still there, but it is important not to trivialise or underestimate the additional complexity of managing another big, complex system. Managed services may help, but in many cases these services are nascent (Amazon EKS was only announced at the end of 2017 for example).

Networking Nightmares

A more traditional model of services running on known hosts, with known addresses, has a fairly simple networking setup.

However, when using microservices, generally there will be many services distributed across many nodes, which typically means there’s going to be a much more complicated networking arrangement. There will be load balancing between services, DNS may be more heavily used, virtual networking layers, etc etc, to attempt to ‘hide’ the complexity of this networking.

However, as per Tesler’s Law (or the Law of Conservation of Compexlity), this networking complexity is inherent - when you are finding real, runtime issues in larger scale clusters, it can often be at a very low networking level. These sorts of issues can be very hard to diagnose. I have started tracking some examples at the end of the article, but I think that Tinder’s Migration to Kuberenetes shows this challenge very well.

Overall - the transition is still likely to be for the best, but doesn’t come without some serious challenges at the networking level, which will require some serious expertise to deal with!

The Death of Microservice Madness!

Avoid the madness by making careful and considered decisions. To help out on this I’ve noted a few questions you might want to ask yourself, and what the answers might indicate:

You can download a PDF copy here: microservice-questions.pdf

Final Thoughts: Don’t Confuse Microservices with Architecture

I’ve deliberately avoided the ‘a’ word in this article. But my friend Zoltan made a very good point when proofing this article (which he has contributed to).

There is no microservice architecture. Microservices are just another pattern or implementation of components, nothing more, nothing less. Whether they are present in a system or not does not mean that the architecture of the system is solved.

Microservices relate in many ways more to the technical processes around packaging and operations rather than the intrinsic design of the system. Appropriate boundaries for components continues to be one of the most important challenges in engineering systems.

Regardless of the size of your services, whether they are in Docker containers or not, you will always need to think carefully about how to put a system together. There are no right answers, and there are a lot of options.

I hope you found this article interesting! As always, please do comment below if you have any questions or thoughts. You can also follow some lively discussions on:

Appendix: Further Reading

The following links might be of interest:

Martin Fowler - Bounded Context - Martin’s articles are great, I’d thoroughly recommend this.
Martin Fowler - Microservices - An often recommended introduction to the pattern.
Microservices - Good or Bad? - Björn Frantzén’s thoughts on microservices, after reading this article.
When Not To Do Microservices - Excellent post on the topic from Christian Posta
Sean Hull - 30 questions to ask a serverless fanboy - Interesting thoughts on the challenges of serverless, from a serverless fan!
Dave Kerr - Monoliths to Microservices - Practical tips for CI/CD and DevOps in the Microservice world - A recent conference presentation I did on devops with microservices.
Alexander Yermakov - Microservices without fundamentals - A response to this article, with Alex’s thoughts and counterpoints to the points raised here (see also Microservices as a self sufficient concept)

Please do share anything else you think makes great reading or watching on the topic!

Thanks

Thanks José from campusmvp.es for having the article translated in Spanish - La muerte de la locura de los microservicios en 2018!

Case Studies

Some interesting examples of experiences I am collecting of larger organisations who have made large scale transitions to microservices:

Tinder’s Move to Kubernetes

References

https://trends.google.com/trends/explore?date=today%205-y&q=microservice ↩︎
If you don’t want to miss the article, you can subscribe to the RSS Feed, or follow me on LinkedIn or Twitter. ↩︎
Docker Compose is a good solution, Fuge is very clever, and there is also the option of running orchestration locally as is the case with something like MiniKube. ↩︎
Google Kubernetes Engine, a managed service from Google Cloud Platform for Kubernetes: https://cloud.google.com/kubernetes-engine/ ↩︎
Amazon Elastic Container Services for Kubernetes, a managed service from Amazon Web Services for Kubernetes: https://aws.amazon.com/eks/ ↩︎

Integrating OpenShift and Splunk for Docker Container Logging

Sun, 29 Oct 2017 07:15:04 +0000

In this article I’m going to show you how to set up OpenShift to integrate with Splunk for logging in a Docker container orchestration environment.

These techniques could easily be adapted for a standard Kubernetes installation as well!

The techniques used in this article are based on the Kubernetes Logging Cluster Administration Guide. I also found Jason Poon’s article Kubernetes Logging with Splunk very helpful.

First, clone the Terraform AWS OpenShift repo:

git clone git@github.com:dwmkerr/terraform-aws-openshift

This repo can be used to create a vanilla OpenShift cluster. I’m adding ‘recipes’ to the project, which will allow you to mix in more features (but still keep the main codebase clean). For now, let’s merge in the ‘splunk’ recipe:

cd terraform-aws-openshift
git pull origin recipes/splunk

Pulling this recipe in adds the extra config and scripts required to set up Splunk¹.

Now we’ve got the code, we can get started!

Create the Infrastructure

To create the cluster, you’ll need to install the AWS CLI and log in, and install Terraform.

Before you continue, be aware: the machines on AWS we’ll create are going to run to about $250 per month:

Once you are logged in with the AWS CLI just run:

make infrastructure

You’ll be asked to specify a region:

Any AWS region will work fine, use us-east-1 if you are not sure.

It’ll take about 5 minutes for Terraform to build the required infrastructure, which looks like this:

Once it’s done you’ll see a message like this:

The infrastructure is ready! A few of the most useful parameters are shown as output variables. If you log into AWS you’ll see our new instances, as well as the VPC, network settings etc etc:

Installing OpenShift

Installing OpenShift is easy:

make openshift

This command will take quite some time to run (sometimes up to 30 minutes). Once it is complete you’ll see a message like this:

You can now open the OpenShift console. Use the public address of the master node (which you can get with $(terraform output master-url)), or just run:

make browse-openshift

The default username and password is admin and 123. You’ll see we have a clean installation and are ready to create our first project:

Close the console for now.

Installing Splunk

You’ve probably figured out the pattern by now…

make splunk

Once this command is complete, you can open the Splunk console with:

make browse-splunk

Again the username and password is admin and 123. You can change the password on login, or leave it:

You can close the Splunk console now, we’ll come back to it shortly.

Demoing Splunk and OpenShift

To see Splunk and OpenShift in action, it helps to have some kind of processing going on in the cluster. You can create a very basic sample project which will spin up two nodes which just write a counter every second as a way to get something running:

make sample

This will create a simple ‘counter’ service:

We can see the logs in OpenShift:

Almost immediately you’ll be able to see the data in Splunk:

And because of the way the log files are named, we can even rip out the namespace, pod, container and id:

That’s it! You have OpenShift running, Splunk set up and automatically forwarding of all container logs. Enjoy!

How It Works

I’ve tried to keep the setup as simple as possible. Here’s how it works.

How Log Files Are Written

The Docker Engine has a log driver which determines how container logs are handled². It defaults to the json-file driver, which means that logs are written as a json file to:

/var/lib/docker/containers/{container-id}/{container-id}-json.log

Or visually:

Normally we wouldn’t touch this file, in theory it is supposed to be used internally¹ and we would use docker logs <container-id>.

In theory, all we need to do is use a Splunk Forwarder to send this file to our indexer. The only problem is that we only get the container ID from the file name, finding the right container ID for your container can be a pain. However, we are running on Kubernetes, which means the picture is a little different…

How Log Files Are Written - on Kubernetes

When running on Kubernetes, things are little different. On machines with systemd, the log driver for the docker engine is set to journald (see Kubernetes - Logging Architecture.

It is possible to forward journald to Splunk, but only by streaming it to a file and then forwarding the file. Given that we need to use a file as an intermediate, it seems easier just to change the driver back to json-file and forward that.

So first, we configure the docker engine to use json-file (see this file):

sed -i '/OPTIONS=.*/c\OPTIONS="--selinux-enabled --insecure-registry 172.30.0.0/16 --log-driver=json-file --log-opt max-size=1M --log-opt max-file=3"' /etc/sysconfig/docker

Here we just change the options to default to the json-file driver, with a max file size of 1MB (and maximum of three files, so we don’t chew all the space on the host).

Now the cool thing about Kubernetes is that it creates symlinks to the log files, which have much more descriptive names:

We still have the original container log, in the same location. But we also have a pod container log (which is a symlink to the container log) and another container log, which is a symlink to the pod container log.

This means we can read the container log, and extract some really useful information from the file name. The container log file name has the following format:

/var/log/containers/{container-id}/{container-id}-json.log

How Log Files Are Read

Now that we are writing the log files to a well defined location, reading them is straightforward. The diagram below shows how we use a splunk-forwarder to complete the picture:

First, we create a DaemonSet, which ensures we run a specific pod on every node.

The DaemonSet runs with a new account which has the ‘any id’ privilege, allowing it to run as root. We then mount the log folders into the container (which are owned by root, which is why our container needs these extra permissions to read the files).

The pod contains a splunk-forwarder container, which is configured to monitor the /var/log/containers folder. It also monitors the docker socket, allowing us to see docker events. The forwarder is also configured with the IP address of the Splunk Indexer.

Footnotes

As a reference, you can also see the recipe pull request to see what changes from a ‘vanilla’ installation to add Splunk: Splunk Recipe Pull Request ↩︎ ↩︎
It is useful to check the documentation on logging drivers for Docker. See Configure Logging Drivers and Docker Log Driver Plugins. It is possible to create custom log drivers. However, at the time of writing only the journald and json-file log drivers will work with the integrated logging view in OpenShift. ↩︎