Golang on dwmkerr.com

Manipulating Istio and other Custom Kubernetes Resources in Golang

Mon, 08 Oct 2018 21:34:02 +0000

In this article I’ll demonstrate how to use Golang to manipulate Kubernetes Custom Resources, with Istio as an example. No knowledge of Istio is needed, I’ll just use it to demonstrate the concepts!

Istio is a highly popular Service Mesh platform which allows engineers to quickly add telemetry, advanced traffic management and more to their service-based applications.

One interesting element of how Istio works is that when deployed into a Kubernetes cluster, many key configuration objects are handled as Custom Resources. Custom Resources are a very powerful Kubernetes feature, which allow you to create your own ‘first class’ resources (just like Pods, ReplicaSets, Deployments or whatever) and then interface with them using kubectl or the Kubernetes APIs.

In this article I’ll show you how to interface with these Custom Resources using the Golang Kubernetes client.

CRDs: A Quick Overview

When you set up Istio for your cluster, one common thing you will likely do is specify how you will route traffic. This can be quite sophisticated, as shown below:

Figure 1: Istio Traffic Management Examples, from istio.io

One way for a system like this to be configured would be to have a ConfigMap which contains the definition of how services are routed.

However, Istio actually registers new types of resources (Custom Resource Definitions) which represent things like Gateways or Services. We can create/update/delete/manipulate them just like any other Kubernetes object.

For example, I could create a virtual service for the example above with something like this:

cat << EOF | kubectl create -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: service2
spec:
 hosts:
 - "*"
 gateways:
 - demo1-gateway
 http:
 - route:
 - destination:
 host: service2
 subset: v1
 weight: 95
 - destination:
 host: service2
 subset: v2
 weight: 5
EOF

Again, the important thing is not the specific content of this resource, more the fact that I can treat my Istio resources just like I would any other Kubernetes object:

$ kubectl get virtualservices.networking.istio.io
NAME AGE
service2 93s

Or:

$ kubectl delete virtualservices.networking.istio.io/service2

I can use edit, describe, register lifecycle events, watch for changes, and so on.

Working with CRDs in Golang

The Golang Kubernetes Client allows you to create strongly defined types which you can then use to interface with CRDs. An example is in the Red Hat blog post Kubernetes Deep Dive: Code Generation for Custom Resources.

This is an excellent approach, but can feel pretty heavy if you want to quickly access some data, and don’t want to have to generate a lot of code.

There is an alternative, which is to use the DynamicClient. The preferred approach seems to be the first, which involves code generation, so little documentation exists for the second approach. However, it is actually very simple.

Here’s an example of how you can list all Istio VirtualService resources, without having to generate any code:

import (
 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/dynamic"
)

// Create a Dynamic Client to interface with CRDs.
dynamicClient, _ := dynamic.NewForConfig(config)

// Create a GVR which represents an Istio Virtual Service.
virtualServiceGVR := schema.GroupVersionResource{
 Group: "networking.istio.io",
 Version: "v1alpha3",
 Resource: "virtualservices",
}

// List all of the Virtual Services.
virtualServices, _ := dynamicClient.Resource(virtualServiceGVR).Namespace("default").List(metav1.ListOptions{})
for _, virtualService := range virtualServices.Items {
 fmt.Printf("VirtualService: %s\n", virtualService.GetName())
}

This snippet omits setup and error-handling for clarity, the full example is in the k8s-list-virtualservices.go gist.

Patching CRDs in Golang

You may have noticed that the .Resource().Namespace().List() code looks very similar to the structure for making API calls when using the Kubernetes Clientset. In fact, it is essentially the same. Looking at the interface, you can see you have all of the operations you’d expect:

Create
Update
Delete
Get

And so on. This is nice because you can use the same trick in my article ‘Patching Kubernetes Resources in Golang’ to manipulate these entities, without ever having to create a structure to represent it.

Here’s another abbreviated example, this time showing how we can adjust the weight of the routing from the services to 50%/50%:

import (
 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/dynamic"
)

// Create a GVR which represents an Istio Virtual Service.
virtualServiceGVR := schema.GroupVersionResource{
 Group: "networking.istio.io",
 Version: "v1alpha3",
 Resource: "virtualservices",
}

// Weight the two routes - 50/50.
patchPayload := make([]PatchUInt32Value, 2)
patchPayload[0].Op = "replace"
patchPayload[0].Path = "/spec/http/0/route/0/weight"
patchPayload[0].Value = 50
patchPayload[1].Op = "replace"
patchPayload[1].Path = "/spec/http/0/route/1/weight"
patchPayload[1].Value = 50
patchBytes, _ := json.Marshal(patchPayload)

// Apply the patch to the 'service2' service.
_, err := dynamicClient.Resource(virtualServiceGVR).Namespace("default").Patch("service2", types.JSONPatchType, patchBytes)

See the full example in the gist k8s-patch-virtualservice.go

After running the sample, you can use the Kubernetes CLI to verify the changes:

$ kubectl get virtualservices.networking.istio.io/service2 -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
clusterName: ""
creationTimestamp: 2018-10-08T09:53:16Z
generation: 0
name: service2
namespace: default
resourceVersion: "487435"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/service2
uid: fac5930c-cadf-11e8-90a2-42010a94005b
spec:
gateways:
- demo1-gateway
hosts:
- '*'
http:
- route:
- destination:
host: service2
subset: v1
weight: 50
- destination:
host: service2
subset: v2
weight: 50

Keep It Simple!

That’s it! This trick made something I was working on a lot easier, but it took a little bit of experimentation to get right. I hope you find the approach useful. Please share any thoughts/questions in the comments.

Patching Kubernetes Resources in Golang

Tue, 24 Jul 2018 06:33:17 +0000

Recently I needed to be able to quickly adjust the number of replicas in a Kubernetes Replication Controller. The original solution I’d seen pulled down the spec, modified it, then updated it. There’s a better way!

There’s a patch API for Kubernetes resources. Patching resources is faster and easier than pulling them and updating the spec wholesale. However, the documentation is a little limited.

After some trial and error I got it working, here’s the solution. I thought it might be helpful to share for others!

The Solution

I’ll start with the solution. If this is all you need, you are good to go. The details of how this works are presented afterwards. In this example I’ll update the number of replicas in the my-rc controller:

package main

import (
 "encoding/json"
 "fmt"

 types "k8s.io/apimachinery/pkg/types"
 "k8s.io/client-go/kubernetes"
 _ "k8s.io/client-go/plugin/pkg/client/auth"
 "k8s.io/client-go/tools/clientcmd"
)

var (
 // Leave blank for the default context in your kube config.
 context = ""

 // Name of the replication controller to scale, and the desired number of replicas.
 replicationControllerName = "my-rc"
 replicas = uint32(3)
)

// patchStringValue specifies a patch operation for a string.
type patchStringValue struct {
 Op string `json:"op"`
 Path string `json:"path"`
 Value string `json:"value"`
}

// patchStringValue specifies a patch operation for a uint32.
type patchUInt32Value struct {
 Op string `json:"op"`
 Path string `json:"path"`
 Value uint32 `json:"value"`
}

func scaleReplicationController(clientSet *kubernetes.Clientset, replicasetName string, scale uint32) error {
 payload := []patchUInt32Value{{
 Op: "replace",
 Path: "/spec/replicas",
 Value: scale,
 }}
 payloadBytes, _ := json.Marshal(payload)
 _, err := clientSet.
 CoreV1().
 ReplicationControllers("default").
 Patch(replicasetName, types.JSONPatchType, payloadBytes)
 return err
}

func main() {
 // Get the local kube config.
 fmt.Printf("Connecting to Kubernetes Context %v\n", context)
 config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
 clientcmd.NewDefaultClientConfigLoadingRules(),
 &clientcmd.ConfigOverrides{CurrentContext: context}).ClientConfig()
 if err != nil {
 panic(err.Error())
 }

 // Creates the clientset
 clientset, err := kubernetes.NewForConfig(config)
 if err != nil {
 panic(err.Error())
 }

 // Scale our replication controller.
 fmt.Printf("Scaling replication controller %v to %v\n", replicationControllerName, replicas)
 err = scaleReplicationController(clientset, replicationControllerName, replicas)
 if err != nil {
 panic(err.Error())
 }
}

This code is also available in the k8s-patch.go gist.

The Mechanism

The Kubernetes Patch API supports a few different methods for modifying resources. It is important to be aware that there is not a universally accepted ‘standard’ approach to representing a change to a resource in a REST API.

There are three strategies you can use to patch:

merge: follows the JSON Merge Patch Spec (RFC 7386)
stragetic: A strategic merge, which addresses some limitations of the merge patch (noted in this doc.
json: follows the JSON Patch Spec (RFC 6902)

These are documented in detail at:

docs/devel/api-conventions.md#patch-operations

The mechanism I’ve used here is json, which I think is the clearest to the reader. To use this strategy we need to build a payload describing what we are changing. This might look like this:

{
 "op": "replace",
 "path": "/spec/replicas",
 "value": 4
}

The op field can be remove, replace, add etc etc (all the details are in the RFC 6902), or the slightly more readable jsonpatch.com). This allows the operation to be very explicit to the reader, which is helpful. We create a struct which represents an operation on a string or integer (or whatever data type we need), serialize it and pass to the API.

Under the hood, the Golang client will simply translate this into an HTTP call which will look like something like this:

PATCH /api/v1/namespaces/default/replicationcontrollers/app-server-blue HTTP/1.1
Host: 127.0.0.1
Content-Type: application/json-patch+json
Content-Length: 70
[{
"op": "replace",
"path": "/spec/replicas",
"value": 4
}]

This corresponds to the documentation on the Patch Operations. Note that the patch operation type is specified in the Content-Type header.

Hopefully this’ll help you if you need to patch resources, are struggling with the docs and are a Go noob like me! Any tips on how to make the code cleaner or more idomatic would be welcome.

Thanks to the following articles and issues which helped me unpick this:

Is it worth persevering with Golang?

Wed, 01 Jun 2016 22:10:40 +0000

I recently decided to try out the Go Programming Language, by building a little project called google-it which let’s me run a google search from a terminal:

The idea behind the project is simple - avoid jumping into a browser if you need to quickly look up something which you can find in the first line of a Google search result. The idea is to try and stay in the zone. For example, forgotten how to split panes in tmux?

google-it "split pane tmux"

Would probably show enough information to get going without leaving the terminal.

Anyway, the project itself is not that useful¹ but it seemed like an ideal project to use as a learning exercise for a new language. After perhaps 10-20 hours of learning, coding and messing around, I’m wondering - is it worth persevering with Golang?

Update 3/6/2016 If you are learning too, check the Tips for Noobs section at the end of the article, great tips from members of the community!

Why Go²?

The decision to choose Go for this learning exercise was fairly arbitrary. For the last few years I’ve been using mainly interpretted languages or languages which use a platform (.NET, Node.js, Java etc) and the idea of going back to something which compiles into in good ol’ binaries seemed appealing. I’d also heard a lot of good things about Go in general, mostly relating to simplicity, ease of use and concurrency.

Why Persevere?

That’s where I’m looking for guidance. I’ve collected some of my observations so far, and my overall experience with the language is uninspiring. Anyone who can comment on what makes Go great, or whether my ambivalence is justified will help me decide whether to build my next mess-around project in Go or move on to something else.

Frustrations So Far

Before I upset anyone, this is all just the opinion of a total Go noob with maybe 15 hours of coding time in Go. But I’ve been developing using a few different languages and platforms for while.

Folder structure is way too opinionated

Setup itself is easy, at least on unix or a Mac. But like many coders, I’m anal-retentive about how I like to organise things:

~
└───repositories
├───github.com
├───dwmkerr
├───project1
├───etc
├───organisation1
├───etc
├───bitbucket.com
├───etc

This is how I structure my projects on all my machines, and it works for me.

Go forces me to put all of my Go projects in the $GOPATH, so now I have:

└───repositories
├───github.com
├───etc
├───go
├───src
├───github.com
├───dwmkerr
├───goproject1

Which unnecessarily spreads out my projects. Other thoughts:

My src folder is increasingly cluttered with dependent modules, making it harder to find my own work.
Even within the project folder, I have little flexibility. I’d like to have a src folder to keep my code in, with just the README.md at the root (leaving space for a docs folder and others if necessary) - this cannot be done, so my root folder is cluttered.
Again, in the project folder itself, I cannot use sub-folders for code. Some might argue if you need subfolders you have too much code in one project.

All in all it feels like there are a lot of constraints for structure and organisation, with little benefit.

Update 3/6/2016 Steve Francia has rightly pointed out that points 2 and 3 are actually wrong, a project can be simply a main.go file in the root and a set of submodules, see this comment for details.

Update 3/6/2016 A very nice way to separate internal and external go modules is described in this reddit thread.

The idiomatic approach to error handling is flawed

This is likely to prove contentious.

My code contains sections like this:

func LoadSettings() (Settings, error) {
var s Settings
exists, err := exists(GetSettingsPath())
if err != nil {
return s, err
}
if !exists {
return CreateDefaultSettings(), nil
}
raw, err := ioutil.ReadFile(GetSettingsPath())
if err != nil {
return s, err
}
json.Unmarshal(raw, &s)
return s, err
}

I see smells:

The s structure is created even though I may not need it.
Even worse, it is returned uninitialised in error conditions.
Repetitive code for dealing with error conditions for calls.

Now I could avoid the first smell by returning a pointer to the structure, but that incures unnecessary complexity and heap allocations. Here I feel the language is forcing me to do something awful (return a structure I know is invalid) and expect the caller to deal with it.

Even worse - the calling code now does this:

settings, err := LoadSettings()
if err != nil {
color.Red("Error loading settings: ", err)
os.Exit(1)
}

I’ve seen this in many places - nested calls passing the same error around, with little extra context, and eventually terminating.

This is what exceptions are for.

Native exceptions in languages handle this for you, giving stack information and killing the process by default.

The ‘pass the error on to the caller approach’ may not be the right way to go, but the Go blog suggests exactly this:

https://blog.golang.org/error-handling-and-go

And to me it stinks a bit. If this is the truly desired idomatic approach, then why not supported it natively by the language? Here’s the same pseudo-code in F#:

let loadSettings =
let path = getSettingsPath()
match exists path with
| true -> path |> readFile |> readSettings
| _ -> createDefaultSettings
match loadSettings with
| Some settings -> // ..whatever
| None -> // ..deal with errors

If loadSettings can’t return settings, it doesn’t return settings. If the caller doesn’t handle the ’no settings’ scenario explicitly, the compiler will complain that there’s a case missing. In this case we have an approach which will warn the coder if they miss something.

Inconsistent Syntax

A small one, but when I’m defining a structure I can do this:

type Link struct {
Id string
Uri string
}

but when I’m returning a structure, I need commas:

return Link{
Id: strconv.Itoa(linkNumber),
Uri: item.Link,
}

I can see the benefit of allowing a comma on the last line, to support quick refactoring, but forcing it seems odd. Why commas for some constructs and not others³?

Also, some more ‘unusual’ syntax (depending on your background) is present, I assume to save space:

something := createAndAssign()
// rather than
var something SomeType
something = assign()

But some space saving constructs such as ternary operators are missing:

// easy- c++, c#, java style
something := condition ? case1() : case2()
// easy- python style
something := case1() if condition else case2() // python
// hard - go style
var something SomeType
if condition {
something = case1()
} else {
something = case2()
}

Difficult Debugging

For C, C++, .NET, Java and many other languages, debugging is pretty straightforward. For Node.js, you can just use the excellent Chrome debugging tools. For Go, it seems like it’s much harder.

In my limited time using the language, I avoided gdb because it looked like a lot of work:

https://golang.org/doc/gdb

I did see some projects like godebug which may ease the process but I was initially surprised by the effort needed to get into debugging.

Commenter Sotirios Mantziaris mentioned that delve provides a nice experience as a debugger, so this would be worth exploring.

Delights So Far

It’s also worth talking about what I’ve liked or loved about Go so far.

Simple Tooling

A project can be nothing more than a single file, go knows how to build and install it. Compare that to Java, where you have a lot of ‘project’ related stuff - Gradle stuff, Ant stuff, Maven stuff, xml project files stuff and it feels much cleaner.

The tooling is intuitive, fast and works well if you are happy living in a terminal.

Fantastic Community

I’ve added this observation just recently, since writing the article I’ve had a huge amount of positive input, describing how to improve my code, better understand Go idioms and where its sweet spots like.

For someone new to a language, the community support is great and will really help people just getting into Go get advice and guidance.

Testing as a First Class Citizen

Testing is built in, which is great. Knowing that you can run go test on a project and have a standard way of executing tests is really quite nice. I love npm test for Node.js projects as it has helped standardise testing as a practice (checkout npm install then npm test).

However, I did have to rely on a library, goconvey, to allow me to write tests in the more BBD structured style which I prefer:

func TestSpec(t *testing.T) {
Convey("The param loader", t, func() {
Convey("Should handle no params", func() {
params, err := ParseParams([]string{})
So(params.ShowHelp.Present, ShouldEqual, false)
So(params.Results.Present, ShouldEqual, false)
So(params.Open.Present, ShouldEqual, false)
So(err, ShouldEqual, nil)
})

But that’s a totally personal thing and I’m sure many others will prefer more ‘vanilla’ tests.

Great Documentation

I’ve found everything I’ve needed so far on Go’s own documentation. The documentation is clean, accessible and seems fairly complete from my limited interactions with it.

Delightful Vim Development Experience

OK - this is not a language feature. But if you are a noob like me moving from this:

to this:

made a big difference. The excellent vim-go plugin gives syntax highlighting and supports some really useful commands. As a learner, regularly running :GoLint is really helping me write more ‘conventional’ Go.

What am I missing?

There are some things I know I haven’t had a chance to look at which may really be demonstrating the best parts of Go:

Concurrency patterns
Performance
Writing web servers
Godoc
Debugging with Delve

Should I continue?

At this stage I’m leaning towards moving on and trying something different, hoping that I’ll come back to Go later. Should I persevere with Go for my next project, would Go enthusiasts suggest so and what sort of project hits the ‘sweet spot’ where Go is a really effective choice?

An interesting comment by a colleague was: “I would say the usual ‘does it change the way you think about programming?’, if yes then persevere, if no then are you going to really leverage Go’s strengths (and find out weaknesses) in your project? If no then either change language or project.” was rather insightful.

Any comments are welcome!

Can you help me get better?

Any pull requests to my project or comments which show where I’ve gone wrong and what I could do to improve my experience and code would be welcome at:

https://github.com/dwmkerr/google-it

Thanks!

Tips for Noobs!

Since publishing this article I’ve collected some useful tips from people who’ve commented or got in touch.

Use gofmt and lint

The tool gofmt will update your code to format it in a conventional go style. This’ll help you keep your code consistent with others’. Using a linter will also help you stay conventional - if you are using vim-go you can run it from vim with :GoLint. Thanks @snoproblem!

Understand where Go is a Ferrari

This I am still working on, and it’s certainly tricky for a noob. But commenter devdungeon pointed out that a project like mine is not a great use case for Go - Go excels at speed and concurrency. Projects where that is key are going to be more inspiring. See this comment for more.

Footnotes

Mainly because you have to sign up for the Google Cloud Platform to get an API key so you can perform searches, as Google have deprecated the free and easy search API. ↩︎
https://www.youtube.com/watch?v=DvijZuvEiQo ↩︎
SnoProblem describes why in this comment ↩︎