Olivier Mauras

In my Part 1 - which I know is already more than a year old now - we saw that we could make it very easy to replicate between a Git repo and another remote automatically using a post-receive hook. We also saw that it wasn't really scalable as it was looping over the remotes.

So what if we have like 20 different remote repositories that we want to keep in sync with our master repository?

Let's use Consul to the rescue!

Main features

• It scales \o/
• Notifies all remotes in near real time of the master modification

Prerequisites

• You've got a Consul cluster available
• All your Git repos are members of the Consul cluster
• You've set the correct ACL to the key you wanna use for the notification - Check my Securing Consul article for that
• Have curl binary available on master repo

The hook

#!/bin/bash

# Get repo name
REPONAME=$(basename $(dirname $(pwd) .git))
CONSUL_TOKEN="9c058400-8cfb-4dd0-ae75-698165a737ac"
CONSUL_URL="http://localhost:8500/v1/kv/git/${REPONAME}/id?token=${CONSUL_TOKEN}"

read oldrev newrev refname

curl -X PUT -d $newrev ${CONSUL_URL}

That's it, nothing more! Indeed if you don't have any ACL to write on the key git/${REPONAME}/id don't even bother with the token.

The consul watch

{
  "watches": [
  {
    "type": "key",
    "key": "git/${REPONAME}/id",
    "handler": "/etc/consul/watch_scripts/${REPONAME_SYNC}.sh"
  },
  {
    "type": "key",
    "key": "git/${ANOTHER_REPONAME}/id",
    "handler": "/etc/consul/watch_scripts/another_sync_script.sh"
  }
  ]
}

With this configuration, a Consul agent started will constantly watch any modification happening on the configured key.
Upon modification, it will execute the configured handler.
You can indeed have multiple keys watched at the same time - Please see Consul watches documentation for more details.

If you had followed my Securing Consul article, it means you're running the Consul agent with an unpriviledged user. handler will thus be executed using this account, think about it when debugging why your handler doesn't work.
Also if you have set ACL on the key, mind to set the acl_token in your Consul agent config.

The ${REPONAME_SYNC} script

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin

ssh-agent bash -c 'ssh-add /home/git/.ssh/<your_key>.rsa; cd /srv/git/${REPONAME}; git fetch origin; git reset --hard origin/master; git clean -fd'

This script is indeed just an example. You can pretty much do what you want in there and do all the Git voodo you need.
This one is just making sure that the master branch of the clone in /srv/git/${REPONAME} is exactly as the master branch on the origin remote - Nothing fancy.
Please note that handlers will be executed at agent start as well so depending of the script you run, a server down restarted could cope with missing key changes and just get synced again.

This somewhat simple setup lets you have all your Git backup repositories and clones get notified when a modification happens on master repository and act according to your needs.
Here on a local network, it takes 1 second for my clones to be in sync with my master repository.

Variants

While I use Consul there, it's indeed possible to mimic the same behaviour with any publish/subscribe system - Redis, MQTT, Saltstack events/reactors, and the list goes on.


As usual, feel free to contact me if you have any questions.