Securing Consul

Consul
Prevent HTTP API from clients
- Completely
- Allow some users
Key/Value store ACLs
- Deny by default
- Add rights per key
Actually running Consul
Wrap it all up

Consul

Consul is an excellent piece of software, really. I don't think I've been this excited by any other software for the last couple of years.
As they state in their Intro page : Consul has multiple components, but as a whole, it is a tool for discovering and configuring services in your infrastructure
Consul is well documented, robust, fast, replicated, datacenter aware, integrates a Key/Value store, etc... And their IRC community is very friendly.

The only major flaw I've found, is that by default, it is not secure enough.
What I mean by not secure enough, is that you must take a good care of how you configure and run the service if you don't wish to let too much opened doors to a harmful user.

Here's a list of threats I've identified - Please mind that a threat for me may very well be a nice feature for you - followed by configuration to circumvent them:

Any node could join your cluster and access sensitive information
Any user on any joined node can access cluster details/information
Any user on any joined node can execute cluster wide remote commands
Any user on any joined node can tamper with the HTTP API and
- Declare potentially harmful events
- Declare watches with potentially harmful events

Note:
All the commands and system details provided in this excerpt are with an EL6 based distribution in mind. Configurations and data paths are purely based by choice.
Please adapt accordingly :)

Binary execution rights and user running the daemon

No surprise here, this is a rule of thumbs for any daemon running on Unix/Linux, privileges must be dropped as much as possible.
Consul doesn't require you to run it as root and that's a good thing.
The thing is, that by using RPC communication, any user on the joined node can run any cluster wide command as well as remote executing command on all your cluster's nodes. It is thus important that any user cannot execute your system wide Consul binary.

Start by creating a new dedicated user without any shell access with a dedicated home directory:

useradd -d /srv/consul -s /sbin/nologin consul

On most distributions it will create a matching default group for your user as well, if not, do so.

Ensure that the home directory can only be accessed by the new consul user and its default group:

chown -R consul: /srv/consul
chmod 750 /srv/consul

Ensure the Consul binary is executable only by root user and consul group:

chown root:consul /usr/sbin/consul
chmod 750 /usr/sbin/consul

Prevent users to execute their own Consul binary

This is a bit out of the Consul scope, and more a matter of user containment, but indeed you must prevent a user to be able to upload and execute his/her own Consul binary.
The two main solutions i think of:

Mount filesystems where local users have write access with noexec flag
Trusted Path Execution from the Grsecurity patchset

Configuration access rights

It's important that you forbid any read access of your Consul configuration by other unprivileged user in order to not leak sensible settings.

mkdir /etc/consul
chown root:consul /etc/consul
chmod 750 /etc/consul

Prevent rogue nodes joining the cluster

As described in this page there's two major way to prevent that any untrusted node join your cluster

Encrypt

With the help of consul keygen command, you can generate gossip encryption key that you can use in your server/agent configuration

{
  "encrypt": "UgmMsZwR2XFNGGvJTnpHRg=="
}

This key must be used on both servers and clients for the communication to work.

TLS

TLS can/must be used to verify servers and clients authenticity. Consul requires that all servers and clients have certificates signed by a certificate authority.
You can leverage the use of your corporate PKI if you have one, just remember as per this thread that Go requires the certificate to have extendedKeyUsage clientAuth enabled.
Here's a sample configuration of enabling TLS:

Servers:

{
  "ca_file": "/etc/consul/ssl/ca_cert.pem",
  "cert_file": "/etc/consul/ssl/server.pem",
  "key_file": "/etc/consul/ssl/server.key",
  "verify_incoming": true,
  "verify_outgoing": true
}

Clients:

{
  "ca_file": "/etc/consul/ssl/ca_cert.pem",
  "cert_file": "/etc/consul/ssl/client.pem",
  "key_file": "/etc/consul/ssl/client.key",
  "verify_outgoing": true
}

Disable remote execution

Remote execution let's you execute commands and get the output of said command using
consul exec "command"
So if you were running your Consul daemon as root, running the command consul exec "shutdown -h now" would have the tremendous effect of shutting down all your Consul servers as well as all the clients nodes joined to them...
There's luckily a configuration option that should in my opinion be enabled by default:

{
  "disable_remote_exec": true
}

Now this remote execution feature could prove itself useful... This is why you limit its availability per node - Consul servers should never have it enabled - as well as its execution rights, by running the Consul binary with the unprivileged consul user we created earlier which you can couple with sudo.

Prevent HTTP API from clients

Consul HTTP API is by default available for all joined nodes on http://localhost:8500, this means that by default anyone connected on a cluster's node can use the API and/or K/V store to gather informations or change settings.

Completely

Plain and simple, use iptables to prevent access to the port:

iptables -A OUTPUT -p tcp -m tcp --dport 8500 -j REJECT

Allow some users

As you may require some of your nodes to be able to access the API and/or the K/V store, you can use the ipt_owner module of iptables:

iptables -A OUTPUT -m tcp -p tcp --dport 8500 -m owner --uid-owner consul -j ACCEPT

This would let only the consul user access the port locally, which is the user under which your daemon is running, and is unavailable to an unprivileged user.

Key/Value store ACLs

I've found the ACL system for the key/value store quite unsettling, it can be broken out like this:

Enable ACL at datacenter level
Set a master token - Not mandatory but i find it easier to manage
Chose a default policy
Set access rules to keys using API

Deny by default

As a habit, i found that denying by default is easier to manage and allow read and/or write accesses per clients.
Enabling the ACL system is a server only configuration:

{
  "acl_datacenter": "<your datacenter name>",
  "acl_master_token": "f45cbd0b-5022-47ab-8640-4eaa7c1f40f1",
  "acl_default_policy": "deny",
  "acl_down_policy": "deny"
}

acl_master_token can be easily generated with the uuidgen command but this is just to keep the same thing as Consul generates, it seems to accept any string format.

Add rights per key

You can list the ACLs using this curl command curl "http://localhost:8500/v1/acl/list?token=f45cbd0b-5022-47ab-8640-4eaa7c1f40f1&pretty=true"

As an example we'll add an ACL giving read only permission to the store key git/lastcommit.

First create a text file containing the following json code:

{                      
  "Name": "git_slave",
  "Type": "client",
  "Rules": "key git/lastcommit {policy = read}"
}

Then create the ACL using your acl_master_token:

curl -X PUT -d @kv_create_rule.json http://localhost:8500/v1/acl/create?token=f45cbd0b-5022-47ab-8640-4eaa7c1f40f1

This curl command will return you a client token that you can reuse in your agent configurations that need to access - through watches - to this key

{
  "acl_token": "d17fade7-2391-45a4-aa00-c9ed5e707e0b"
}

Actually running Consul

Here's an EL6 based init script to automatically start Consul with the correct user at startup:

#!/bin/sh
#
# consul        Start the consul daemon
#
# Author:      Olivier Mauras <olivier@mauras.ch>
#
# chkconfig: 345 99 10
# description: Starts the Consul daemon
#
# processname: consul

# Source function library.
. /etc/rc.d/init.d/functions

RETVAL=0

# Default variables
CONSUL_BIN="/usr/sbin/consul"
CONSUL_CONF="/etc/consul"
CONSUL_USER="consul"

# See how we were called.
case "$1" in
  start)
        echo -n "Starting Consul daemon: "
        daemon --check consul --user $CONSUL_USER "$CONSUL_BIN agent -config-dir=$CONSUL_CONF > /dev/null 2>&1 &"
        echo
        ;;
  stop)
        echo -n "Stopping Consul daemon: "
        killproc consul
        echo
        ;;
  status)
        status consul
        RETVAL=$?
        ;;
  restart)
        $0 stop
        $0 start
        RETVAL=$?
        ;;
  *)
        echo "Usage: consul {start|stop|status|restart}"
        exit 1
esac

exit $REVAL

Wrap it all up

I've made up an RPM package, that takes care of creating the consul user and setting up the correct rights, on the binary - It also includes the init script :). You can download the SRPM directly or just use my el6 extras repository.

Use your favorite configuration management tool - Rudder? :) - to handle the configuration deployment and /etc/consul rights.

Don't hesitate to contact me if you find inconsistencies, or if you have questions about this guide.