Terraform on dwmkerr.com

Dynamic and Configurable Availability Zones in Terraform

Tue, 11 Dec 2018 21:24:34 +0000

When building Terraform modules, it is a common requirement to want to allow the client to be able to choose which region resources are created in, and which availability zones are used.

I’ve seen a few ways of doing this, none of which felt entirely satisfactory. After a bit of experimentation I’ve come up with a solution which I think really works nicely. This solution avoids having to know in advance how many availability zones we’ll support.

To demonstrate, I’ve set up a module which deploys a cluster of web servers. My goal is to be able to configure the region, VPC CIDR block, subnets and subnet CIDR blocks as below:

module "cluster" {
source = "github.com/dwmkerr/terraform-aws-vpc"
# Note how we can specify any number of availability zones here...
region = "ap-northeast-2"
vpc_cidr = "10.0.0.0/16"
subnets = {
ap-northeast-2a = "10.0.1.0/24"
ap-northeast-2b = "10.0.2.0/24"
ap-northeast-2c = "10.0.3.0/24"
}
# This just defines the number of web servers to deploy, and uses
# adds my public key so I can SSH into the servers...
web_server_count = "3"
public_key_path = "~/.ssh/id_rsa.pub"
}

The example module is at github.com/dwmkerr/terraform-aws-vpc. Let’s take a look at some of the key elements.

The Variables

We define the required variables very explicitly, with descriptions and a variable type to avoid confusion:

variable "region" {
description = "The region to deploy the VPC in, e.g: us-east-1."
type = "string"
}
variable "vpc_cidr" {
description = "The CIDR block for the VPC, e.g: 10.0.0.0/16"
type = "string"
}
variable "subnets" {
description = "A map of availability zones to CIDR blocks, which will be set up as subnets."
type = "map"
}

The VPC

Now that we have defined the variables, we can set up the VPC:

// Define the VPC.
resource "aws_vpc" "cluster" {
cidr_block = "${var.vpc_cidr}"
enable_dns_hostnames = true
}
// An Internet Gateway for the VPC.
resource "aws_internet_gateway" "cluster_gateway" {
vpc_id = "${aws_vpc.cluster.id}"
}
// Create one public subnet per key in the subnet map.
resource "aws_subnet" "public-subnet" {
count = "${length(var.subnets)}"
vpc_id = "${aws_vpc.cluster.id}"
cidr_block = "${element(values(var.subnets), count.index)}"
map_public_ip_on_launch = true
depends_on = ["aws_internet_gateway.cluster_gateway"]
availability_zone = "${element(keys(var.subnets), count.index)}"
}
// Create a route table allowing all addresses access to the IGW.
resource "aws_route_table" "public" {
vpc_id = "${aws_vpc.cluster.id}"
route {
cidr_block = "0.0.0.0/0"
gateway_id = "${aws_internet_gateway.cluster_gateway.id}"
}
}
// Now associate the route table with the public subnet - giving
// all public subnet instances access to the internet.
resource "aws_route_table_association" "public-subnet" {
count = "${length(var.subnets)}"
subnet_id = "${element(aws_subnet.public-subnet.*.id, count.index)}"
route_table_id = "${aws_route_table.public.id}"
}

There are a few things of interest here. First, we can easily build a variable number of subnets by using the count field on the aws_subnet resource:

resource "aws_subnet" "public-subnet" {
count = "${length(var.subnets)}"
availability_zone = "${element(keys(var.subnets), count.index)}"
cidr_block = "${element(values(var.subnets), count.index)}"
}

By using the Terraform Interpolation Syntax, and in particular the count, keys, values and element functions, we can grab the subnet name and CIDR block from the variables.

The Web Server Cluster

A cluster of web servers behind a load balancer are created by the module, to demonstrate that it works. There is little of interest in the script except for how the subnets are referenced:

resource "aws_autoscaling_group" "cluster_node" {
name = "cluster_node"
vpc_zone_identifier = ["${aws_subnet.public-subnet.*.id}"]
launch_configuration = "${aws_launch_configuration.cluster_node.name}"
}

Note that we can specify the entire list of subnet ids by using the * symbol in the resource path - ["${aws_subnet.public-subnet.*.id}"].

That’s It!

That’s really all there is to it. I quite like this approach. I think it makes it very clear what is going on with the infrastructure, and is fairly manageable.

One question which may be raised is why I am not using the cidrsubnet function to automatically calculate the CIDR blocks for the subnets. The reason is purely one of preference - I prefer to explicitly specify the CIDR blocks and use various patterns to set conventions. For example, if I see an IP address such as 10.0.3.121 then it is in the third AZ of my public subnet, or 10.2.2.11 is in the second AZ of my locked down data zone.

You can see a sample Terraform module which uses this pattern at: github.com/dwmkerr/terraform-aws-vpc-example. This module also has a basic build pipeline and is published on the Terraform Registry. I’ll also be updating my AWS Openshift module to use this pattern.

Integrating OpenShift and Splunk for Docker Container Logging

Sun, 29 Oct 2017 07:15:04 +0000

In this article I’m going to show you how to set up OpenShift to integrate with Splunk for logging in a Docker container orchestration environment.

These techniques could easily be adapted for a standard Kubernetes installation as well!

The techniques used in this article are based on the Kubernetes Logging Cluster Administration Guide. I also found Jason Poon’s article Kubernetes Logging with Splunk very helpful.

First, clone the Terraform AWS OpenShift repo:

git clone git@github.com:dwmkerr/terraform-aws-openshift

This repo can be used to create a vanilla OpenShift cluster. I’m adding ‘recipes’ to the project, which will allow you to mix in more features (but still keep the main codebase clean). For now, let’s merge in the ‘splunk’ recipe:

cd terraform-aws-openshift
git pull origin recipes/splunk

Pulling this recipe in adds the extra config and scripts required to set up Splunk¹.

Now we’ve got the code, we can get started!

Create the Infrastructure

To create the cluster, you’ll need to install the AWS CLI and log in, and install Terraform.

Before you continue, be aware: the machines on AWS we’ll create are going to run to about $250 per month:

Once you are logged in with the AWS CLI just run:

make infrastructure

You’ll be asked to specify a region:

Any AWS region will work fine, use us-east-1 if you are not sure.

It’ll take about 5 minutes for Terraform to build the required infrastructure, which looks like this:

Once it’s done you’ll see a message like this:

The infrastructure is ready! A few of the most useful parameters are shown as output variables. If you log into AWS you’ll see our new instances, as well as the VPC, network settings etc etc:

Installing OpenShift

Installing OpenShift is easy:

make openshift

This command will take quite some time to run (sometimes up to 30 minutes). Once it is complete you’ll see a message like this:

You can now open the OpenShift console. Use the public address of the master node (which you can get with $(terraform output master-url)), or just run:

make browse-openshift

The default username and password is admin and 123. You’ll see we have a clean installation and are ready to create our first project:

Close the console for now.

Installing Splunk

You’ve probably figured out the pattern by now…

make splunk

Once this command is complete, you can open the Splunk console with:

make browse-splunk

Again the username and password is admin and 123. You can change the password on login, or leave it:

You can close the Splunk console now, we’ll come back to it shortly.

Demoing Splunk and OpenShift

To see Splunk and OpenShift in action, it helps to have some kind of processing going on in the cluster. You can create a very basic sample project which will spin up two nodes which just write a counter every second as a way to get something running:

make sample

This will create a simple ‘counter’ service:

We can see the logs in OpenShift:

Almost immediately you’ll be able to see the data in Splunk:

And because of the way the log files are named, we can even rip out the namespace, pod, container and id:

That’s it! You have OpenShift running, Splunk set up and automatically forwarding of all container logs. Enjoy!

How It Works

I’ve tried to keep the setup as simple as possible. Here’s how it works.

How Log Files Are Written

The Docker Engine has a log driver which determines how container logs are handled². It defaults to the json-file driver, which means that logs are written as a json file to:

/var/lib/docker/containers/{container-id}/{container-id}-json.log

Or visually:

Normally we wouldn’t touch this file, in theory it is supposed to be used internally¹ and we would use docker logs <container-id>.

In theory, all we need to do is use a Splunk Forwarder to send this file to our indexer. The only problem is that we only get the container ID from the file name, finding the right container ID for your container can be a pain. However, we are running on Kubernetes, which means the picture is a little different…

How Log Files Are Written - on Kubernetes

When running on Kubernetes, things are little different. On machines with systemd, the log driver for the docker engine is set to journald (see Kubernetes - Logging Architecture.

It is possible to forward journald to Splunk, but only by streaming it to a file and then forwarding the file. Given that we need to use a file as an intermediate, it seems easier just to change the driver back to json-file and forward that.

So first, we configure the docker engine to use json-file (see this file):

sed -i '/OPTIONS=.*/c\OPTIONS="--selinux-enabled --insecure-registry 172.30.0.0/16 --log-driver=json-file --log-opt max-size=1M --log-opt max-file=3"' /etc/sysconfig/docker

Here we just change the options to default to the json-file driver, with a max file size of 1MB (and maximum of three files, so we don’t chew all the space on the host).

Now the cool thing about Kubernetes is that it creates symlinks to the log files, which have much more descriptive names:

We still have the original container log, in the same location. But we also have a pod container log (which is a symlink to the container log) and another container log, which is a symlink to the pod container log.

This means we can read the container log, and extract some really useful information from the file name. The container log file name has the following format:

/var/log/containers/{container-id}/{container-id}-json.log

How Log Files Are Read

Now that we are writing the log files to a well defined location, reading them is straightforward. The diagram below shows how we use a splunk-forwarder to complete the picture:

First, we create a DaemonSet, which ensures we run a specific pod on every node.

The DaemonSet runs with a new account which has the ‘any id’ privilege, allowing it to run as root. We then mount the log folders into the container (which are owned by root, which is why our container needs these extra permissions to read the files).

The pod contains a splunk-forwarder container, which is configured to monitor the /var/log/containers folder. It also monitors the docker socket, allowing us to see docker events. The forwarder is also configured with the IP address of the Splunk Indexer.

Footnotes

As a reference, you can also see the recipe pull request to see what changes from a ‘vanilla’ installation to add Splunk: Splunk Recipe Pull Request ↩︎ ↩︎
It is useful to check the documentation on logging drivers for Docker. See Configure Logging Drivers and Docker Log Driver Plugins. It is possible to create custom log drivers. However, at the time of writing only the journald and json-file log drivers will work with the integrated logging view in OpenShift. ↩︎

Get up and running with OpenShift on AWS

Thu, 02 Feb 2017 07:47:00 +0000

OpenShift is Red Hat’s platform-as-a-service offering for hosting and scaling applications. It’s built on top of Google’s popular Kubernetes system.

Getting up and running with OpenShift Online is straightforward, as it is a cloud hosted solution. Setting up your own cluster is a little more complex, but in this article I’ll show you how to make it fairly painless.

The repo for this project is at: github.com/dwmkerr/terraform-aws-openshift.

Creating the Infrastructure

OpenShift has some fairly specific requirements about what hardware it runs on¹. There’s also DNS to set up, as well as internet access and so on.

All in all, for a bare-bones setup, you’ll need something like this:

Which is (deep breath):

A network
A public subnet, with internet access via a gateway
A master host, which will run the OpenShift master
A pair of node hosts, which will run additional OpenShift nodes
A hosted zone, which allows us to configure DNS
A bastion, which allows us to SSH onto hosts, without directly exposing them
Some kind of basic log aggregation, which I’m using CloudWatch for

This is not a production grade setup, which requires redundant masters and so on, but it provides the basics.

Rather than setting this infrastructure up by hand, this is all scripted with Terraform. To set up the infrastructure, clone the github.com/dwmkerr/terraform-aws-openshift repo:

$ git clone git@github.com:dwmkerr/terraform-aws-openshift
...
Resolving deltas: 100% (37/37), done.

Then use the terraform CLI² to create the infrastructure:

$ cd terraform-aws-openshift/
$ terraform get && terraform apply

You’ll be asked for a region, to deploy the network into, here I’m using us-west-1:

After a few minutes the infrastructure will be set up:

A quick glance at the AWS console shows the new hosts we’ve set up:

The next step is to install OpenShift.

Installing OpenShift

There are a few different ways to install OpenShift, but the one we’ll use is called the ‘advanced installation³’. This essentially involves:

Creating an ‘inventory’, which specifies the hosts OpenShift will be installed on and the installation options
Downloading the advanced installation code
Running the advanced installation Ansible Playbook

To create the inventory, we just run:

sed "s/\${aws_instance.master.public_ip}/$(terraform output master-public_ip)/" inventory.template.cfg > inventory.cfg

This takes our ‘inventory template⁴’ and populates it with the public IP of our master node, which is recorded in a Terraform output variable.

We can then copy the inventory to the bastion:

ssh-add ~/.ssh/id_rsa
scp ./inventory.cfg ec2-user@$(terraform output bastion-public_dns):~

We can again use the Terraform output variables, this time to get the bastion IP. Finally, we pipe our install script to the bastion host:

cat install-from-bastion.sh | ssh -A ec2-user@$(terraform output bastion-public_dns)

There’s a bug which means you might see ansible-playbook: command not found, if so, just run the script again. The install script clones the installation scripts and runs them, using the inventory we’ve provided:

This’ll probably take about 10 minutes to run. And that’s it, OpenShift is installed:

open "https://$(terraform output master-public_dns):8443"

Hit ‘advanced’ and continue, as we’re using a self-signed certificate most browsers will complain:

Enter any username and password (the system is configured to allow anyone to access it by default) and you’ll be presented with the OpenShift console:

As the setup requires three t2.large instances, which are not available on the free plan, you might want to clean up when you are done with:

terraform destroy

Wrapping Up

Hopefully you’ve found this useful, there are more details and references on the README of the github repo:

https://github.com/dwmkerr/terraform-aws-openshift

Comments and feedback are always welcome!

See https://docs.openshift.org/latest/install_config/install/prerequisites.html#system-requirements ↩︎
Use ‘brew install terraform’, full instructions in the README.md ↩︎
See https://docs.openshift.org/latest/install_config/install/advanced_install.html ↩︎
See https://github.com/dwmkerr/terraform-aws-openshift/blob/master/inventory.template.cfg ↩︎

Creating a Resilient Consul Cluster for Docker Microservice Discovery with Terraform and AWS

Mon, 09 Jan 2017 07:10:40 +0000

In this article I’m going to show you how to create a resilient Consul cluster, using Terraform and AWS. We can use this cluster for microservice discovery and management. No prior knowledge of the technologies or patterns is required!

The final code is at github.com/dwmkerr/terraform-consul-cluster. Note that it has evolved somewhat since the time of writing, see the Appendices at the end of the article for details.

Consul, Terraform & AWS

Consul is a technology which enables Service Discovery¹, a pattern which allows services to locate each other via a central authority.

Terraform is a technology which allows us to script the provisioning of infrastructure and systems. This allows us to practice the Infrastructure as Code pattern. The rigour of code control (versioning, history, user access control, diffs, pull requests etc) can be applied to our systems.

And why AWS? We need to create many servers and build a network to see this system in action. We can simulate parts of this locally with tools such as Vagrant, but we can use the arguably most popular² IaaS platfom for this job at essentially zero cost, and learn some valuable skills which are readily applicable to other projects at the same time.

A lot of what we will learn is not really AWS specific - and the Infrastructure as Code pattern which Terraform helps us apply allows us to apply these techniques easily with other providers.

The Goal

The goal is to create a system like this:

In a nutshell:

We have a set of homogenous Consul nodes
The nodes form a cluster and automatically elect a leader
The nodes span more than one availability zone, meaning the system is redundant and can survive the failure of an entire availability zone (i.e. data centre)
The Consul UI is available to view via a gateway
We have two example microservices which register themselves on the cluster, so we can actually see some registered services in the console

As a quick caveat, in reality this setup would typically live in a private subnet, not directly accessible to the outside work except via public facing load balancers. This adds a bit more complexity to the Terraform setup but not much value to the walk-though. A network diagram of how it might look is below, I invite interested readers to try and move to this model as a great exercise to cement the concepts!

Step 1 - Creating our Network

The first logical step is to create the network itself. This means:

The network (in AWS terminology, this is a VPC or Virtual Private Cloud)
The ‘public’ subnet, which defines our IP ranges for hosts
The internet gateway, which provides an entry/exit point for traffic from/to the internet
The firewall rules, which define what traffic can come in and out of the network

All together, that’s this:

Our solution will be made more resilient by ensuring we host our Consul nodes across multiple availability zones³

Creating a VPC and building a subnet is fairly trivial if you have done some network setup before or spent much time working with AWS, if not, you may be a little lost already. There’s a good course on Udemy⁴ which will take you through the process of setting up a VPC which I recommend if you are interested in this, as it is quite hands on. It’ll also show you how to build a more ‘realistic’ network, which also contains a private subnet and NAT, but that’s beyond the scope of this write-up. Instead, I’ll take you through the big parts.

The Network

We’re using AWS, we need to create a VPC. A VPC is a Virtual Private Cloud. The key thing is that it is isolated. Things you create in this network will be able to talk to each other if you let them, but cannot communicate with the outside world, unless you specifically create the parts needed for them to do so.

A private network is probably something you regularly use if you work in a company⁵. Most companies have their own internal network - when you use a computer on that network it can talk to other company computers (such as the company mail server). When you are off that network, you might not be able to access your company email (unless it is publicly available, like gmail, or over a VPN [and by accessing a VPN, you are actually joining the network again, albeit remotely]).

Perhaps the most immediately obvious part of a VPC is that you control the IP addresses. You specify the range of IP addresses which are available to give to machines on the network. When a machine joins, it is given an IP in that range. I’m not going to go into too much detail here, if you are interested let me know and I’ll write up an article on VPCs in detail!

Here’s how I’d suggest scripting AWS infrastructure with Terraform if you haven’t done this before.

Use the AWS console to create what you want
Search the Terraform documentation for the entity you want to create (e.g. VPC), script the component and apply the provisioning
Compare the hand-made VPC to the script-made VPC, if the two are the same, you are done
If the two are different, check the documentation and try again

Ensure you have an AWS account, and note your Secret Key and Access Key. We’ll need these to remotely control it. Here’s the terraform script to create a VPC:

// Setup the core provider information.
provider "aws" {
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
region = "${var.region}"
}
// Define the VPC.
resource "aws_vpc" "consul-cluster" {
cidr_block = "10.0.0.0/16" // i.e. 10.0.0.0 to 10.0.255.255
enable_dns_hostnames = true
tags {
Name = "Consul Cluster VPC"
Project = "consul-cluster"
}
}

This script uses Terraform Variables, such as var.access_key, which we keep in a variables.tf file. Terraform will use the default values defined in the file if they are present, or ask the user to supply them. Let’s build the network:

terraform apply

After supplying the values for the variables, Terraform will provision the network, using the AWS SDK internally.

You’ll see lots of info about what it is creating, then a success message.

The Public Subnet

You don’t put hosts directly into a VPC, they need to go into a structure called a ‘subnet’, which is a part of a VPC. Subnets get their own subset of the VPC’s available IP addresses, which you specify.

Subnets are used to build zones in a network. Why would you need this? Typically it is to manage security. You might have a ‘public zone’ in which all hosts can be accessed from the internet, and a ‘private zone’ which is inaccessible directly (and therefore a better location for hosts with sensitive data). You might have an ‘operator’ zone, which only sysadmins can access, but they can use to get diagnostic information.

Here’s a common subnet layout for multi-tiered applications:

The defining characteristics of zones is that they are used to create boundaries to isolate hosts. These boundaries are normally secured by firewalls, traversed via gateways or NATs etc. We’re going to create two public subnets, one in each of the availability zones⁵:

// Create a public subnet for each AZ.
resource "aws_subnet" "public-a" {
vpc_id = "${aws_vpc.consul-cluster.id}"
cidr_block = "10.0.1.0/24" // i.e. 10.0.1.0 to 10.0.1.255
availability_zone = "ap-southeast-1a"
map_public_ip_on_launch = true
}
resource "aws_subnet" "public-b" {
vpc_id = "${aws_vpc.consul-cluster.id}"
cidr_block = "10.0.2.0/24" // i.e. 10.0.2.0 to 10.0.1.255
availability_zone = "ap-southeast-1b"
map_public_ip_on_launch = true
}

With Terraform, resources can depend on each other. In this case, the subnets need to reference the ID of the VPC we want to place them in (so we use aws_vpc.consul-cluster.id).

The Internet Gateway, Route Tables and Security Groups

The final parts of the network you can see in the ./infrastructure/network.tf script. These are the Internet Gateway, Route Table and Security Group resources. Essentially they are for controlling access between hosts and the internet. AWS have a good guide if you are not familiar with these resources; they don’t add much to the article so I’ll leave you to explore on your own.

That’s it for the network, we now have the following structure:

If you want to see the code as it stands now, check the Step 1 branch. Now we need to look at creating the hosts to install Consul on.

Step 2 - Creating the Consul Hosts

The Consul documentation recommends running in a cluster or 3 or 5 nodes⁶. We want to set up a system which is self-healing - if we lose a node, we want to create a new one.

Enter Auto-Scaling Groups. Auto-scaling groups allow us to define a template for an instance, and ask AWS to make sure there are always a certain number of these instances. If we lose an instance, a new one will be created to keep the group at the correct size⁷.

So we now need to create:

A ‘Launch Configuration’ which determines what instances our Auto-scaling Group creates
A ‘user data script’ which runs on newly created instances, which must install and start Consul
An Auto-scaling group, configured to run five instances across the two public subnets
A load balancer, configured to pass incoming requests for the Consul Admin console to the nodes

Or visually:

Let’s get to it.

The Launch Configuration & Auto-scaling Group

The Launch Configuration will define the characteristics of our instances and the auto-scaling group determines the size of our cluster:

// Launch configuration for the consul cluster auto-scaling group.
resource "aws_launch_configuration" "consul-cluster-lc" {
name_prefix = "consul-node-"
image_id = "${lookup(var.ami_ecs_optimised, var.region)}"
instance_type = "t2.micro"
security_groups = ["${aws_security_group.consul-cluster-vpc.id}"]
lifecycle {
create_before_destroy = true
}
}
// Auto-scaling group for our cluster.
resource "aws_autoscaling_group" "consul-cluster-asg" {
name = "consul-asg"
launch_configuration = "${aws_launch_configuration.consul-cluster-lc.name}"
min_size = 5
max_size = 5
vpc_zone_identifier = [
"${aws_subnet.public-a.id}",
"${aws_subnet.public-b.id}"
]
lifecycle {
create_before_destroy = true
}
}

A few key things to note:

I have omitted the tag properties in the scripts for brevity
The ‘image’ for the launch configuration is looked up based on the region we’ve specified - we’re a basic linux image⁸
We are using micro instances, which are free-tier eligible
The auto-scaling group spans both availability zones.

Once we run terraform apply, we’ll see our auto-scaling group, which references the new launch configuration and works over multiple availability zones:

We can also see the new instances:

These instances don’t do much yet though, we’ve not installed Docker or Consul.

Installing Consul and Accessing the Admin Interface

To set up our instances we use a ‘userdata’ script’ A userdata runs once when an instance is created. We can create a script in our repository, and reference it in our Terraform files.

We add a new file called consul-node.sh to a files folder. This script installs Docker and runs Consul:

yum install -y docker
usermod -a -G docker ec2-user
service docker start

# Get my IP address.
IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
echo "Instance IP is: $IP"

# Start the Consul server.
docker run -d --net=host \
 --name=consul \
 consul agent -server -ui \
 -bind="$IP" \
 -client="0.0.0.0" \
 -bootstrap-expect="1"

Here’s a breakdown of what we’re doing:

Install Docker. These scripts run as root, so we add the ec2-user to the Docker group, meaning when we log in later on via SSH, we can run Docker
Get our IP address. AWS provide a magic address (169.254.169.254) which lets you query data about your instance, see Instance Metadata & User Metadata
Run the Consul docker image in server mode, with the UI enabled, expecting only one instance

The actual scripts contains more! Getting userdata scripts right, testing and debugging them is tricky. See how I do it in detail in Appendix 1: Logging.

Now we need to tell Terraform to include this script as part of the instance metadata. Here’s how we do that:

resource "aws_launch_configuration" "consul-cluster-lc" {
/// ...add the line below....
user_data = "${file("files/consul-node.sh")}"
}

When Consul is running with the -ui option, it provides an admin UI. You can try it by running Consul locally with docker run -p8500:8500 consul and navigating to http://localhost:8500/ui.

We can install a load balancer in front of our auto-scaling group, to automatically forward incoming traffic to a host. Here’s the config:

resource "aws_elb" "consul-lb" {
name = "consul-lb-a"
security_groups = [
"${aws_security_group.consul-cluster-vpc.id}",
"${aws_security_group.web.id}"
]
subnets = [
"${aws_subnet.public-a.id}",
"${aws_subnet.public-b.id}"
]
listener {
instance_port = 8500
instance_protocol = "http"
lb_port = 80
lb_protocol = "http"
}
health_check {
healthy_threshold = 2
unhealthy_threshold = 2
timeout = 3
target = "HTTP:8500/ui/"
interval = 30
}
}

Blow-by-blow:

Create a load balancer, with the same security groups as the rest of the VPC, but also a security group which allows web access
Point to two subnets first subnet
Forward HTTP 8500 traffic
Configure a healthcheck⁹

The final change we make is to add an outputs.tf file, which lists all of the properties Terraform knows about which we want to save. All it includes is:

output "consul-dns" {
value = "${aws_elb.consul-lb.dns_name}"
}

When we finally run terraform apply, we see the public DNS of our load balancer:

And running in a browser on port 8500 we see the Consul admin interface:

Every time we refresh we will likely see a different node. We’ve actually created five clusters each of one node - what we now need to do is connect them all together into a single cluster of five nodes.

If you want to see the code as it stands now, check the Step 2 branch.

Step 3 - Creating the Cluster

Creating the cluster is now not too much of a challenge. We will update the userdata script to tell the consul process we are expecting 5 nodes (via the bootstrap-expect flag.

Here’s the updated script:

# Get my IP address.
IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
echo "Instance IP is: $IP"
# Start the Consul server.
docker run -d --net=host \
--name=consul \
consul agent -server -ui \
-bind="$IP" \
-client="0.0.0.0" \
-bootstrap-expect="5"

The problem is this won’t work… We need to tell each node the address of another server in the cluster. For example, if we start five nodes, we should tell nodes 2-5 the address of node 1, so that the nodes can discover each other.

The challenge is how do we get the IP of node 1? The IP addresses are determined by the network, we don’t preset them so cannot hard code them. Also, we can expect nodes to occasionally die and get recreated, so the IP addresses of nodes will in fact change over time.

Getting the IP addresses of nodes in the cluster

There’s a nice trick we can use here. We can ask AWS to give us the IP addresses of each host in the auto-scaling group. If we tell each node the addresses of the other nodes, then they will elect a leader themselves¹⁰.

There are a couple of things we need to do to get this right. First, update the userdata script to provide the IPs of other nodes when we’re starting up, then update the role of our nodes so that they have permissions to use the APIs we’re going to call.

Getting the Cluster IPs

This is actually fairly straightforward. We update our userdata script to the below:

# A few variables we will refer to later...
ASG_NAME=consul-asg
REGION=ap-southeast-1
EXPECTED_SIZE=5

# Return the id of each instance in the cluster.
function cluster-instance-ids {
 # Grab every line which contains 'InstanceId', cut on double quotes and grab the ID:
 # "InstanceId": "i-example123"
 #....^..........^..^.....#4.....^...
 aws --region="$REGION" autoscaling describe-auto-scaling-groups \
 --auto-scaling-group-name $ASG_NAME \
 | grep InstanceId \
 | cut -d '"' -f4
}

# Return the private IP of each instance in the cluster.
function cluster-ips {
 for id in $(cluster-instance-ids)
 do
 aws --region="$REGION" ec2 describe-instances \
 --query="Reservations[].Instances[].[PrivateIpAddress]" \
 --output="text" \
 --instance-ids="$id"
 done
}

# Wait until we have as many cluster instances as we are expecting.
while COUNT=$(cluster-instance-ids | wc -l) && [ "$COUNT" -lt "$EXPECTED_SIZE" ]
do
 echo "$COUNT instances in the cluster, waiting for $EXPECTED_SIZE instances to warm up..."
 sleep 1
done

# Get my IP address, all IPs in the cluster, then just the 'other' IPs...
IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
mapfile -t ALL_IPS < <(cluster-ips)
OTHER_IPS=( ${ALL_IPS[@]/${IP}}/} )
echo "Instance IP is: $IP, Cluster IPs are: ${CLUSTER_IPS[@]}, Other IPs are: ${OTHER_IPS[@]}"

# Start the Consul server.
docker run -d --net=host \
 --name=consul \
 consul agent -server -ui \
 -bind="$IP" \
 -retry-join="${OTHER_IPS[0]}" -retry-join="${OTHER_IPS[1]}" \
 -retry-join="${OTHER_IPS[2]}" -retry-join="${OTHER_IPS[3]}" \
 -bootstrap-expect="$EXPECTED_SIZE"

Right, here’s what’s going on:

We create a few variables we’ll use repeatedly
We create a cluster-instance-ids function which returns the ID of each instance in the auto-scaling group
We create a cluster-ips function which returns the private IP address of each instance in the cluster.
We wait until the auto-scaling group has our expected number of instances (it can take a while for them all to be created)
We get the 5 IP addresses
We remove our IP from the array, leaving us with the IPs of the other nodes
We start the Consul agent in server mode, expecting 5 nodes and offering the IP of each other agent

The problem is, if we try to run the script we will fail, because calling the AWS APIs requires some permissions we don’t have. Let’s fix that.

Creating a Role for our nodes

Our nodes now have a few special requirements. They need to be able to query the details of an auto-scaling group and get the IP of an instance¹¹.

We will need to create a policy which describes the permissions we need, create a role, attach the policy to the role and then ensure our instances are assigned the correct role. This is consul-node-role.tf file:

// This policy allows an instance to discover a consul cluster leader.
resource "aws_iam_policy" "leader-discovery" {
name = "consul-node-leader-discovery"
path = "/"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1468377974000",
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeAutoScalingGroups",
"ec2:DescribeInstances"
],
"Resource": [
"*"
]
}
]
}
EOF
}
// Create a role which consul instances will assume.
// This role has a policy saying it can be assumed by ec2
// instances.
resource "aws_iam_role" "consul-instance-role" {
name = "consul-instance-role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
// Attach the policy to the role.
resource "aws_iam_policy_attachment" "consul-instance-leader-discovery" {
name = "consul-instance-leader-discovery"
roles = ["${aws_iam_role.consul-instance-role.name}"]
policy_arn = "${aws_iam_policy.leader-discovery.arn}"
}
// Create a instance profile for the role.
resource "aws_iam_instance_profile" "consul-instance-profile" {
name = "consul-instance-profile"
roles = ["${aws_iam_role.consul-instance-role.name}"]
}

Terraform is a little verbose here! Finally, we update our launch configuration to ensure that the instances assume this role.

resource "aws_launch_configuration" "consul-cluster-lc" {
// Add this line!!
iam_instance_profile = "${aws_iam_instance_profile.consul-instance-profile.id}"
}
}

Let’s create the cluster again, with terraform apply. When we log into the UI we should now see a cluster containing all five nodes:

This code is all in the Step 3 branch.

If you are familiar with Consul, this may be all you need. If not, you might be interested in seeing how we actually create a new instance to host a service, register it with Consul and query its address.

Step 4 - Adding a Microservice

I’ve created a docker image for as simple a microservice as you can get. It returns a quote from Futurama’s Zapp Brannigan. The image is tagged as dwmkerr/zapp-service.

On a new EC2 instance, running in either subnet, with the same roles as the Consul nodes, we run the following commands:

# Install Docker
sudo su
yum update -y
yum install -y docker
service docker start
# Get my IP and the IP of any node in the server cluster.
IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
NODE_ID=$(aws --region="ap-southeast-1" autoscaling describe-auto-scaling-groups --auto-scaling-group-name "consul-asg" \
| grep InstanceId \
| cut -d '"' -f4 \
| head -1)
NODE_IP=$(aws --region="ap-southeast-1" ec2 describe-instances \
--query="Reservations[].Instances[].[PrivateIpAddress]" \
--output="text" \
--instance-ids="$NODE_ID")
# Run the consul agent.
docker run -d --net=host \
consul agent \
-bind="$IP" \
-join=$NODE_IP
# Run registrator - any Docker images will then be auto registered.
docker run -d \
--name=registrator \
--net=host \
--volume=/var/run/docker.sock:/tmp/docker.sock \
gliderlabs/registrator:latest \
consul://localhost:8500
# Run the example microservice - registrator will take care of letting consul know.
docker run -d -p 5000:5000 dwmkerr/zapp-service

What’s going on here?

We grab our own IP address and the IP address of the first instance we find in the server cluster, using the same tricks as before
We run the Consul agent - telling it the IP to use to join the cluster
We run Registrator, a handy utility which will automatically register any new services we run to Consul
We run a goofy sample microservice (which registrator will register for us)

Now we can check the Consul UI:

And there we have it. Our new node joins the cluster (as a client), we can register a new service with Consul.

We can call this service from any node in the subnet, seeing output like the below:

In this example, I used a DNS SRV query to ask where the zapp-service is, was told it was at 10.0.2.158 on port 5000, then called the service, receiving a response. I can discover any service using this method, from any node. As services are added, removed, moved etc, I can ask Consul for accurate information on where to find them.

Check the Step 4 branch to see the code in its current state.

Step 5 - Spanner Throwing

We can now try to throw some spanners in the works, to see how resilient the system is.

According to the Deployment Table from the Consul documentation, a cluster of five nodes means we have a quorum of three nodes (i.e. a minimum of three nodes are needed for a working system). This means we can tolerate the failure of two nodes.

The easiest way to test this is to simply manually kill two nodes:

If we pick two random nodes, as above, and terminate them, we see the cluster determines that we have two failed nodes but will still function (if one was the leader, a new leader will be automatically elected):

What’s nice about this setup is that no manual action is needed to recover. Our load balancer will notice the nodes are unhealthy and stop forwarding traffic. Our auto-scaling group will see the nodes have terminated and create two new ones, which will join the cluster in the same way as the original nodes. Once they join, the load balancer will find them healthy and bring them back into rotation.

We can see from the load balancer monitoring that it notices we have unhealthy nodes and also notices when new ones come into service:

A quick check of the admin dashboard shows we now have a recovered system, with five healthy nodes:

The nodes which were terminated are still listed as failing. After 72 hours Consul will stop trying to periodically reconnect to these nodes and completely remove them¹².

Wrapping Up

Hopefully this should provide a good starting point to think about building your own resilient and robust systems for services like Consul.

Interesting areas to look into to extend the project would be:

Setting up alerts so that if we lose more than one node, we are informed
Automating resilience tests by programatically bringing down servers and monitoring how long it takes the system to return to five nodes
Instead of using a userdata script to set up a node, bake it into a new custom AMI with Packer
Adding alerts for if we lose three of more nodes, which always requires manual intervention (see Outage Recovery)

As always, any questions or comments are welcome! All code is available at github.com/dwmkerr/terraform-consul-cluster.

Appendix 1: Logging

Small typos or mistakes in the userdata script are almost impossible to effectively diagnose. The scripts were actually built in the following way:

Draft a script on my local machine which configures script logging and CloudWatch¹³
Spin up a new EC2 instance manually
SSH onto the instance, and run the script line by line until I’m sure it’s right
Ensure the logs are forwarded to CloudWatch, then add the more complex features and repeatedly test

I’ve included CloudWatch logging in the code. In this write-up I’ve omitted this code as it is purely for diagnostics and doesn’t contribute to the main topic. The setup is in the consul-node.sh and consul-node-role.tf files.

If you want more details, let me know, or just check the code. I would heartily recommend setting up logging like this for all but the most straightforward projects:

Being able to diagnose issues like this is vital when working with distributed systems which may be generating many log files.

Appendix 2: Modularisaton

I got some a great PR from arehmandev which modularises the code. This makes it more reusable and cleans up the structure significantly. If you want to see the before/after, check the original PR at https://github.com/dwmkerr/terraform-consul-cluster/pull/4.

Footnotes

Further Reading

This kind of pattern is critical in the world of microservices, where many small services will be running on a cluster. Services may die, due to errors or failing hosts, and be recreated on new hosts. Their IPs and ports may be ephemeral.It is essential that the system as a whole has a registry of where each service lives and how to access it. Such a registry must be resilient, as it is an essential part of the system. ↩︎
Most popular is a fairly loose term. Well ranked by Gartner and anecdotally with the largest infrastructure footprint. https://www.gartner.com/doc/reprints?id=1-2G2O5FC&ct=150519&st=sb ↩︎
This is AWS parlance again. An availabilty zone is an isolated datacenter. Theoretically, spreading nodes across AZs will increase resilience as it is less likely to have catastrophic failures or outages across multiple zones. ↩︎
I don’t get money from Udemy or anyone else for writing anything on this blog. All opinions are purely my own and influenced by my own experience, not sponsorship. Your milage may vary (yada yada) but I found the course quite good: https://www.udemy.com/aws-certified-solutions-architect-associate/. ↩︎
For more expert readers that may sound horribly patronising, I don’t mean it to be. For many less experienced technologists the basics of networking might be more unfamiliar! ↩︎ ↩︎
See https://www.consul.io/docs/internals/consensus.html. ↩︎
A common pattern is to actually make the group size dynamic, responding to events. For example, we could have a group of servers which increases in size if the average CPU load of the hosts stays above 80% for five minutes, and scales down if it goes below 10% for ten minutes. This is more common for app and web servers and not needed for our system. ↩︎
Specifically, the current latest Amazon Linux AMI. ↩︎
Check the admin UI every 30 seconds, more than 3 seconds indicates a timeout and failure. Two failures in a row means an unhealthy host, which will be destroyed, two successes in a row for a new host means healthy, which means it will receive traffic. ↩︎
This is a fairly sophisticated topic in itself, see Consul - Consensus Protocol for details. ↩︎
In fact, we actually have more permissions required, because in the ‘real’ code we also have logs forwarded to CloudWatch. ↩︎
These nodes can be removed manually, see Consul Force Leave. ↩︎
Amazon’s service for managing and aggregating logs ↩︎