Olivier Mauras

Introducing dns-blackhole

2017-09-05T23:34:00+02:00

So the other day I wanted to remove ads on my Android phone, but apparently you can't install extensions on Android chromium and I didn't feel like using Firefox for that.
Since I use non rooted CopperHeadOS as my main mobile OS, solutions like AdAway that require root access were not possible either.... What to do then?
Oh wait don't we have a recursive DNS resolver on this OpenVPN tunnel? We sure do! And wouldn't it be nice that it removes ads for us? Indeed it would!

On this particular setup I was using PowerDNS Recursor and so stumbled upon this nice blog article from last year where they explain how to use the Disconnect list used by the Mozilla Focus project to efficiently block ads/tracking domains.

While the process is quite easy, after reading some of the project readme it seemed even more simple to just get the domains from the original list.
I thus wrote this tiny script that would do the job for me, without even touching the Git repo from the blog post, and PowerDNS Recursor was now happily blocking DNS queries to some advertisement and tracking domains...
Some seemed suddenly too few... With the ~1500 domains that the Disconnect list was offering, I was only blocking a tiny fraction of the trash that most of Internet has become today. We could surely do better :)

I decided that it would be interesting to include the content of several well known host lists and ended up with more than 690000 domains blocked!!
That sure was some improvement, but this proved the lua method to be unable to cope with that many entries - recursor crashed at startup with a constant overflow - so I searched how to make recursor behave correctly while still be able to block all those bad domains.
The idea has been to generate a zone file that would respect recursor's syntax and that could be fed through the forward-zones-file option.
Syntax looks like this:

<domain>=<where to forward the query>

The nice thing here is that if you don't specify any <where to forward the query>, recursor will directly reply with a NXDOMAIN answer.
That's perfect! For all those bad domains my client would receive a NXDOMAIN return and wouldn't even try to connect anywhere. Exactly what was needed!

So basically if we can make a file with one domain per line for PowerDNS Recursor, we should be able to do so for other DNS servers...

After a major cleanup, here comes dns-blackhole!

Features

Not bound to a specific DNS server, generates a file format of your choice
Supports 3 different list format
- Host file
- Easylist
- Disconnect
Lets you whitelist/blacklist domains
YAML configuration file
Easy installation with pip

It has been successfully tested with:

Even generating an agregated host file is possible.

The project's README covers pretty much every details needed to use it, just keep in mind that without a whitelist your browsing sessions will lack a lot of content.
Here's a short whitelist example:

amazon.com
amazon.de
amazon.fr
amazonaws.com
cloudfront.net
freenode.net
google.ch
google.com
google.fr
googleapis.com
imgur.com
intel.com
licdn.com
linkedin.com
msecnd.net # Skype CDN
netdna-cdn.com # bootstrapcdn.com
reddit.com
redditstatic.com
search.ch
thetvdb.com
yahoo.com
yimg.com
youtube.com

And indeed you'll have to re-run dns-blackhole each time you add something to the whitelist.

I hope you'll find this tool useful, and as usual feel free to contact me should you have any questions.

devops REX 2016

2016-12-22T11:40:00+01:00

Corrected slides are available on slideshare

Hope that we get in touch again next year :)

2016-12-05T22:30:00+01:00

Git replication: Part 2 - Now you can scale

2016-12-01T20:00:00+01:00

Main features
Prerequisites
The hook
The consul watch
The ${REPONAME_SYNC} script
Variants

In my Part 1 - which I know is already more than a year old now - we saw that we could make it very easy to replicate between a Git repo and another remote automatically using a post-receive hook. We also saw that it wasn't really scalable as it was looping over the remotes.

So what if we have like 20 different remote repositories that we want to keep in sync with our master repository?

Let's use Consul to the rescue!

Main features

It scales \o/
Notifies all remotes in near real time of the master modification

Prerequisites

You've got a Consul cluster available
All your Git repos are members of the Consul cluster
You've set the correct ACL to the key you wanna use for the notification - Check my Securing Consul article for that
Have curl binary available on master repo

The hook

#!/bin/bash

# Get repo name
REPONAME=$(basename $(dirname $(pwd) .git))
CONSUL_TOKEN="9c058400-8cfb-4dd0-ae75-698165a737ac"
CONSUL_URL="http://localhost:8500/v1/kv/git/${REPONAME}/id?token=${CONSUL_TOKEN}"

read oldrev newrev refname

curl -X PUT -d $newrev ${CONSUL_URL}

That's it, nothing more! Indeed if you don't have any ACL to write on the key git/${REPONAME}/id don't even bother with the token.

The consul watch

{
  "watches": [
  {
    "type": "key",
    "key": "git/${REPONAME}/id",
    "handler": "/etc/consul/watch_scripts/${REPONAME_SYNC}.sh"
  },
  {
    "type": "key",
    "key": "git/${ANOTHER_REPONAME}/id",
    "handler": "/etc/consul/watch_scripts/another_sync_script.sh"
  }
  ]
}

With this configuration, a Consul agent started will constantly watch any modification happening on the configured key.
Upon modification, it will execute the configured handler.
You can indeed have multiple keys watched at the same time - Please see Consul watches documentation for more details.

If you had followed my Securing Consul article, it means you're running the Consul agent with an unpriviledged user. handler will thus be executed using this account, think about it when debugging why your handler doesn't work.
Also if you have set ACL on the key, mind to set the acl_token in your Consul agent config.

The ${REPONAME_SYNC} script

#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin

ssh-agent bash -c 'ssh-add /home/git/.ssh/<your_key>.rsa; cd /srv/git/${REPONAME}; git fetch origin; git reset --hard origin/master; git clean -fd'

This script is indeed just an example. You can pretty much do what you want in there and do all the Git voodo you need.
This one is just making sure that the master branch of the clone in /srv/git/${REPONAME} is exactly as the master branch on the origin remote - Nothing fancy.
Please note that handlers will be executed at agent start as well so depending of the script you run, a server down restarted could cope with missing key changes and just get synced again.

This somewhat simple setup lets you have all your Git backup repositories and clones get notified when a modification happens on master repository and act according to your needs.
Here on a local network, it takes 1 second for my clones to be in sync with my master repository.

Variants

While I use Consul there, it's indeed possible to mimic the same behaviour with any publish/subscribe system - Redis, MQTT, Saltstack events/reactors, and the list goes on.

As usual, feel free to contact me if you have any questions.

New Alpine Linux repository, EL6 repositories discontinued

2016-06-19T12:45:00+02:00

I've decided to start a new Alpine Linux repository, mainly to support latest Grsecurity with options that are disabled in defaut Alpine's kernel configuration.
All sources and configurations can be found in their dedicated git repository.

At the same time, I've decided to discontinue the EL6 repositories as I prefer to focus my time on Alpine. I removed all binary RPMs but SRPMS will stay available.

Please check repositories page for details.

As usual feel free to contact me if you encounter any issues.

Kickstart: Securely save your Rudder/CFEngine's agent keys during installation

2016-06-08T14:00:00+02:00

Reinstalling a server, be it for testing or upgrade, I'm sure you've had the need to.
If you use Rudder/CFEngine, you know you need to take care of those little agent keys if you don't want to have problems with your configuration management. Indeed you could end up deleting your node in your Rudder interface, but this is taking you a bit away of a fully automated process isn't it?
I've noted that the less I mess up with my agent keys, the less I need to do manual tasks, so it would be nice if I could ensure a server keeps the same agent key for as long as it exists.

Now the main issue is that the kickstart file concerning a server is publicly available as it must be fetched by the machine when it starts its installation - One could try to put some address restrictions in place, but I felt like it would be a burden... I'm not sure I'll be on the correct subnet next time I need to verify a Kickstart file for a server - that means that the saving process will be in clear for everyone that gets access to the kickstart file.
Having someone getting access to a server agent key could have dramatic results, so the saving process must be as secure as possible.

To summarize:

Must be easy to save/retrieve the key during Kickstart process - No need to fire up a restore from a backup agent or anything, it's fully automated
Saving process logic is in clear for everyone
Secure enough that anyone can't just retrieve your server key
Server identifier must be unique so that keys can be retrieved easily across reinstallations
Lowest dependancy possible during Kickstart - curl, sha512sum
It's fine for Root user to be able to retrieve the saved server keys - If you're root on the server you can already access its CFEngine keys...

I've decided to use the system-uuid that is contained in /sys/class/dmi/id/product_uuid. You can find it as well using dmidecode or lshw as Root.
Mind you that depending of your setup, this may not work for you - especially on VMs where you could get duplicates if they're not centrally managed.
Warning: Older distributions that still use HAL may let this information leak to non privileged users through lshal. It requires HAL to be installed and the service to be started though ... Beware of default installations!

Now that I have my system UUID, I'll make a hash of it - a long one - and upload/retrieve my keys from an HTTP server using my hash as filename/prefix.
Why over HTTP ? Simply because it requires nothing else than curl to do so, and that it can be hosted pretty much anywhere you want. For that very specific need, I've made a tiny HTTP file server that only does:

Stores a file uploaded by its name
Cannot overwrite a file that already exists
Download a file by its name
Returns 404 if filename is unknown

Please find below the Cobbler snippet that will do just that.

#set $rudder_cf_path    = '/var/rudder/CFEngine-community'
#set $rudder_opt_path   = '/opt/rudder'
#set $rudder_keys_path  = $rudder_cf_path + '/ppkeys'

# Generate server unique hash
SRV_UUID=\$(cat /sys/class/dmi/id/product_uuid)
SRV_HASH=\$(echo -n \$SRV_UUID | sha512sum | cut -f 1 -d " ")

# Check if our hash exist
curl -X GET https://$http_store:8080/get/\$SRV_HASH.pub > /tmp/localhost.pub
grep "404 page not found" /tmp/localhost.pub &> /dev/null
RET=\$?

# Process result
if [ \$RET == 0 ]; then
    # Keys doesn't exist upload them
    curl -s -X POST https://$http_store:8080/add/\$SRV_HASH.priv -F "file=@$rudder_keys_path/localhost.priv"
    curl -s -X POST https://$http_store:8080/add/\$SRV_HASH.pub -F "file=@$rudder_keys_path/localhost.pub"
    curl -s -X POST https://$http_store:8080/add/\$SRV_HASH.hive -F "file=@$rudder_opt_path/etc/uuid.hive"
else
    # They seem to exist download them 
    curl -s -X GET https://$http_store:8080/get/\$SRV_HASH.priv > $rudder_keys_path/localhost.priv
    curl -s -X GET https://$http_store:8080/get/\$SRV_HASH.pub > $rudder_keys_path/localhost.pub
    curl -s -X GET https://$http_store:8080/get/\$SRV_HASH.hive > $rudder_opt_path/etc/uuid.hive
fi

So with this code you generate a hash that is never printed anywhere in the kickstart file nor on the installed system but only gets saved/retrieved remotely on/from a HTTP file server.
This example uses Rudder default path but by using Cobbler you could easily set the path based on a profil ksmeta to make it available for CFEngine as well.

#set $ppkeys_path = $getVar('$ppkeys_path', '/var/lib/cfengine/ppkeys')

The code will store files looking like this on your HTTP file server:

0bc29c7d604b38c14c42b495c075e003e2b1cc36f92d60ad70eb9c5c2408175ff72d6038dfca6a2a1fe1a8c4032642650295a4bc7dab8d57177942092c83e78b.hive
0bc29c7d604b38c14c42b495c075e003e2b1cc36f92d60ad70eb9c5c2408175ff72d6038dfca6a2a1fe1a8c4032642650295a4bc7dab8d57177942092c83e78b.priv
0bc29c7d604b38c14c42b495c075e003e2b1cc36f92d60ad70eb9c5c2408175ff72d6038dfca6a2a1fe1a8c4032642650295a4bc7dab8d57177942092c83e78b.pub
906e84086789e32421a45622286e096a735c36cd636934bc91dcb8c280a8b4424958452903199bb133330721402fb12bbf175c78ea706d4f54d089dc15567863.hive
906e84086789e32421a45622286e096a735c36cd636934bc91dcb8c280a8b4424958452903199bb133330721402fb12bbf175c78ea706d4f54d089dc15567863.priv
906e84086789e32421a45622286e096a735c36cd636934bc91dcb8c280a8b4424958452903199bb133330721402fb12bbf175c78ea706d4f54d089dc15567863.pub

So that even you shouldn't know which file comes from which server.

Now your reinstallation should be as simple as wiping your disk boot sectors and reboot.

As usual, feel free to contact me if you have any question!

Systemd: Run it last ...

2016-02-26T06:00:00+01:00

You remember the good ol' days when running a command last was as easy as

echo "last_command" >> /etc/rc.local

Sure you do, and I bet you also know that now that systemd has become the default init on pretty much all major distributions, it doesn't work anymore.
Wait, /etc/rc.local still works and a lot of people will jump on the occasion to tell you, but NO, it doesn't work to run your command last. It's even written there in the file

#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local

Yes that means that putting command there will have them run in parallel with the other services and you have no control over it.

Did it need improvement? Well maybe, I particularly like how openrc has handled the local service. Let's look at the README in /etc/local.d

This directory should contain programs or scripts which are to be run
when the local service is started or stopped.

If a file in this directory is executable and it has a .start extension,
it will be run when the local service is started. If a file is
executable and it has a .stop extension, it will be run when the local
service is stopped.

All files are processed in lexical order.

Keep in mind that files in this directory are processed sequencially,
and the local service is not considered started or stopped until
everything is processed, so if you have a process which takes a long
time to run, it can delay your boot or shutdown processing.

Seems like a smart way of enhancing the feature to me.

But let's get back to our concern, how does one run a command last on systemd?
When you reach to people which are supposedly knowledgeable about systemd, you often end up with this kind of discussion:

you: Hello, I'm wondering what would be the best way to paint my car red?
them: Why don't you first start telling us why you think you need to paint your car red
you: ??? I want it red that's all ???
them: Well usually people think they _want_ their car red, but that's wrong, they just _need_ it black
you: Well I don't _need_ it black I _want_ it red
them: Yep, you _want_ it black

Yes, true story, people will try to make you think that you don't need/want to run anything last. Well aren't we using Linux because we like the idea of being able to do what we want with our systems? Seems like mentality is changing...

Come on there must be a way?! Yes there is, and this is how you do:

Create a new custom target
Make this target Requires multi-user.target
Create a unit file for your command with After multi-user.target
Put the unit file in custom.target.wants
Make this new custom target default

Sounds easy right? Here's a bit more details.

Create a new custom target

/etc/systemd/system/custom.target

[Unit]
Description=Custom Target
Requires=multi-user.target
After=multi-user.target
AllowIsolate=yes

Here you have a new target that should run after multi-user.target that can be used in a similar way as a runlevel thanks to AllowIsolate

Create a unit file for your command

[Unit]
Description=My last command
After=multi-user.target

[Service]
Type=simple
ExecStart=/sbin/last_command

[Install]
WantedBy=custom.target

After is important here else it'll be run in parallel with other services of multi-user.target. You wanna put this service file in /etc/systemd/system/custom.target.wants directory.

Make your new target default

systemctl list-units --type target --all

Should show you the complete list of available targets, with your custom.target being inactive

systemctl isolate custom.target

Will switch your current target to your custom.target. Good time to see if it's working and debug.

ln -sf /etc/systemd/system/custom.target /etc/systemd/system/default.target

Will switch your custom target to the default.

You can now reboot and enjoy your "last_command" being run last, but note that services that take time to start, like a Java app, may very well still be starting when "last_command" will be executed.
If this is an issue you'll have to play with ExecStartPre=/ExecStartPost=

When I have to do this to simply run a command last in my boot process, I clearly miss the good ol' days, and luckily there's still a bunch a excellent distributions that keep the KISS principle as a priority.

As usual, feel free to contact me!

Rudder 3.2: Default group classes

2016-02-16T06:00:00+01:00

If you've followed the latest Rudder's release note, you've noted in 3.2 changelog - amongst other nice stuff - this new feature:

Add CFEngine classes for each group of a node

Now in Rudder 3.2 you have this new display in the group page:

In this example you have two available CFengine classes that you can use in your mustache templates - Or actually anywhere you need to match them:

group_d57ff100_33c7_410f_82a5_a6098b6de321
group_test1

As the group name can change, you also have the possibility to use the group UUID.

I believe you've already read this nice post on Normation's blog about the usage of mustache templates with NCF.
Let's see an example with /etc/resolv.conf

{{vars.parameters.rudder_header}}
search domain.com
options timeout:1
options attempts:2
{{#classes.group_test1}}
nameserver 192.168.1.10
nameserver 192.168.1.11
{{/classes.group_test1}}
{{#classes.group_test2}}
nameserver 10.10.0.10
nameserver 10.10.0.11
{{/classes.group_test2}}

Group classes are already defined by Rudder now, it's just up to you to have them in your templates.

As usual, feel free to contact me if you have any question!

Speed up Rudder new node's detection - 2016 Update

2016-02-16T06:00:00+01:00

In my previous article, I was proposing a way to update the new inventory list sent to Rudder more quickly.
Upon upgrading to Rudder 3.2 and CentOS 7 at the same time, this was not working anymore. It seemed that the way inventories are handled slightly changed and for some reasons inotify wasn't detecting the inventories creation anymore.
After some digging, it appears that the inventories are renamed from a temporary .davfs.xxxxx instead of being created directly in incoming directory - Seems like a webdav behaviour change in Apache 2.4.

Please find below and updated version using IN_MOVED_TO event to correctly detect new inventories sent to server - I've kept IN_CREATE event, but it shouldn't be matched anymore, you can safely remove it.

#!/usr/bin/python
import os, datetime, subprocess, re, thread, sys
import pyinotify

inventories_path    = "/var/rudder/inventories/"
cmd_new_inv_to_cmdb = "/opt/rudder/bin/rudder agent run -b sendInventoryToCmdb"

wm = pyinotify.WatchManager() # Watch Manager
mask = pyinotify.IN_DELETE | pyinotify.IN_CREATE | pyinotify.IN_MOVED_TO 

def exec_sendtocmdb(file, dummy1):
    process = subprocess.Popen(cmd_new_inv_to_cmdb, stdout=subprocess.PIPE, stderr=None, shell=True)
    output = process.communicate()
    date = datetime.datetime.now()
    print("%s: %s - %s\n" % (date, file, output[0]))

class EventHandler(pyinotify.ProcessEvent):
    def process_IN_CREATE(self, event):
        date = datetime.datetime.now()
        print("%s: %s - %s created\n" % (date, event.path, event.name))

        if "incoming" in event.path:
            if ".davfs" not in event.name:
                dummy_tup = event.name, 'null'
                thread.start_new_thread(exec_sendtocmdb, dummy_tup)

    def process_IN_MOVED_TO(self, event):
        date = datetime.datetime.now()
        print "%s: %s - %s moved to" % (date, event.path, event.name)

        if "incoming" in event.path:
            if ".davfs" not in event.name:
                dummy_tup = event.name, 'null'
                thread.start_new_thread(exec_sendtocmdb, dummy_tup)

    def process_IN_DELETE(self, event):
        date = datetime.datetime.now()
        print("%s: %s - %s deleted\n" % (date, event.path, event.name))

handler = EventHandler()
notifier = pyinotify.Notifier(wm, handler)
wdd = wm.add_watch(inventories_path, mask, rec=True)

notifier.loop()

And as we're now using systemd, here is the unit file to have it running as a service and be automatically respawned in case it should crash.

[Unit]
Description=Rudder inventory watcher
After=syslog.target network.target

[Service]
Type=simple
Environment=PYTHONUNBUFFERED=true
ExecStart=/sbin/rudder-inventory-watcher
PrivateTmp=true
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

As usual, feel free to contact me if you have any question!

Introducing g2rudder

2015-11-11T20:00:00+01:00

Accessing a REST API without the proper tools is often bothersome, especially when you only have a shell and curl at hand - like in a kickstart %post part - and parsing JSON replies or building parameters can quickly become a mess.
g2rudder is here to help you cope with that issue in the form of a gateway written in Go that let's you use Rudder's API easily in a consistent way.

Features

Support Rudder server over HTTPS
Can disable SSL verification and/or provide a root CA certificate
Groups:
- Add hosts in Rudder groups
- Automatically add regex if provided host contains the following characters: *[]|
Hosts:
- Check if hostname is pending
- Check if hostname exists
- Get node uuid from hostname
- Accept pending node from hostname
- Add/delete properties

Usage

Groups:
curl -X POST http://localhost:8181/group/addHost -d "g_uuid=5af30b6b-c8cd-4405-9d0c-410c2b05e220&host=test.domain.com"

Hosts:
curl -X GET http://localhost:8181/host/isExist?host=test.domain.com
curl -X GET http://localhost:8181/host/isPending?host=test.domain.com
curl -X GET http://localhost:8181/host/getId?host=test.domain.com
curl -X POST http://localhost:8181/host/addProperty -d "host=demo-latest.rudder-project.org&key=KEY1&value=VALUE1"
curl -X POST http://localhost:8181/host/delProperty -d "host=demo-latest.rudder-project.org&key=KEY1"
curl -X POST http://localhost:8181/host/accept -d "host=demo-latest.rudder-project.org"

Project readme will be updated along with the addition of new features.
Don't hesitate to contact me if you encounter any issue.

EL6 extras update and new package: Nginx and Nginx-modsecurity

2015-06-26T14:35:00+02:00

Nginx - 1.8.0
- Update to new stable release
- Enable http_auth_request_module
- Compile with GCC 4.9.1 from devtoolset-3
Nginx-modsecurity - 1.8.0
- Update to new stable release
- Enable http_auth_request_module
- Add modsecurity module - 2.9.0
- Compile with GCC 4.9.1 from devtoolset-3

No reason for those versions to not be identical, minor release may change on -modsecurity if modification is only on ModSecurity.
As usual, you can install the package using my el6_extras repository, and feel free to contact me if you encounter any issues.

Git replication: Part 1 - The poor man's way

2015-04-18T05:00:00+02:00

There's a lot of ressources about that on the web, it's a well covered topic already.
Still I wanted to give out my own way of keeping Git repositories synchronized.

Main features:

It's cheap :)
It doesn't scale
Supports pushing/deleting branches and tags
Uses Git remote to know the remote repositories, no config file needed

Prerequisites:

You've set your remotes correctly in your - bare - repository
You've set your SSH key(s) as needed

Installation:

cp <the following script> <your Git repo>/hooks/post-receive && chmod +x <your Git repo>/hooks/post-receive

The hook:

#!/bin/bash
# Read commit values
read oldrev newrev refname
echo "post-receive"

# Is the action a delete ?
if [ $newrev == "0000000000000000000000000000000000000000" ]; then
    PUSH="push --delete"
else
    PUSH="push"
fi

# Get out if there's no remote set
REMOTE=$(git remote)
[[ -z $REMOTE ]] && echo "!! No remote set" && exit 0

# Else loop and push to them
for remote in ${REMOTE}; do
    git $PUSH $remote $refname
done

So you get it as it's looping over the remotes, having more than 2 or 3 will seriously make long pushes... Still works like a charm to keep an external copy of your repository.

EL6 extras updates: Nginx, Uwsgi and GeoIP

2015-04-16T20:00:00+02:00

Nginx - 1.6.3
Uwsgi - 2.0.10
GeoIP - 1.6.5

As usual, you can install the package using my el6_extras repository, and feel free to contact me if you encounter any issues.

Consul update - 0.5.0

2015-02-23T20:00:00+01:00

Consul - 0.5.0

As usual, you can install the package using my el6_extras repository, and feel free to contact me if you encounter any issues.

Grsecurity update - 3.14.33-100

2015-02-19T12:30:00+01:00

3.14.33-100
- Update to grsecurity-3.0-3.14.33-201502181906
- Upgrade to devtoolset 3 with GCC 4.9.1
- Add overlayfs patch from openwrt
- Include kernel config in tarball Diff from the previous version on my Git repo

The biggest change is actually how I now handle the branches for package tagging on the Git repo.
Up until now package tagging couldn't reflect any configuration change and the overrall numbering was then not so consistent. I'm now adding a branch -coredumb wich contains modifications on top of the Grsecurity patchset - Like overlayfs patch addition - as well as the full kernel configuration. All package tags will now be based on a -coredumb branch.

WARNING:

At least on some servers - CPU related ? - pcid detection is still broken...

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Update: Nginx + Uwsgi + Cgit

2015-02-05T01:36:00+01:00

Update:

Steeve Chailloux notified me that this howto wasn't updated to the latest release of the uwsgi package that now have all modules as plugins.
You should read the following:

Install the packages

Now that you're good with the repository, install the packages:

# yum -y install nginx uwsgi uwsgi-cgi cgit

Configure Uwsgi

Just add this simple conf

# mkdir /etc/uwsgi
# cat << EOF > /etc/uwsgi.d/cgit.ini
[uwsgi]
plugin = cgi
socket = 127.0.0.1:8080
uid = cgit
gid = cgit
processes = 1
threads = 2
cgi = /usr/libexec/cgit/cgit.cgi
EOF

There's no init script provided in my package yet, so just manually run - or any way you want

# uwsgi --plugins-dir /usr/lib64/uwsgi/ --ini /etc/uwsgi.d/cgit.ini &

Original post:

As you've seen in my previous post, there's been some package updates in the EL6 Extras repository, so let's have some fun with them!

First you indeed need my EL6 Extras repository if it's not done yet, please start with the repositories page.

Install the packages

Now that you're good with the repository, install the packages:

# yum -y install nginx uwsgi cgit

Configure Cgit

The default config from the package is pretty sane, the only thing you'll need to add to your /etc/gitrc - And indeed your Git repositories config - is:

virtual-root=/

This will keep cgit from adding cgit.cgi to your URL.
To enable syntax highlighting, uncomment the following line in your /etc/cgitrc

source-filter=/usr/libexec/cgit/filters/syntax-highlighting.py

Then install pygments package

# yum -y install python-pygments

Configure Uwsgi

Just add this simple conf

# mkdir /etc/uwsgi
# cat << EOF > /etc/uwsgi/cgit.ini
[uwsgi]
socket = 127.0.0.1:8080
uid = cgit
gid = cgit
processes = 1
threads = 2
cgi = /usr/libexec/cgit/cgit.cgi
EOF

There's no init script provided in my package yet, so just manually run - or any way you want

# uwsgi --ini /etc/uwsgi/cgit.ini &

Configure Nginx

Finally just add this simple server block to your Nginx config and customize it to your needs

server {
    # Change this to your taste
    listen       80 default_server;
    server_name  _;

    # We want to serve cgit static content from /usr/share/cgit and let Nginx cache it
    location ~* ^.+(cgit.(css|png)|favicon.ico) {
        root /usr/share/cgit;
        expires 30d;
    }

    # Now forward everything else to our uwsgi instance
    location / {
        include uwsgi_params;
        uwsgi_modifier1 9;
        uwsgi_pass 127.0.0.1:8080;
    }
}

Start Nginx, and enjoy your Cgit running on your server_name

As usual, contact me if you have any question.

Securing Consul

2015-02-02T23:00:00+01:00

Consul
Prevent HTTP API from clients
- Completely
- Allow some users
Key/Value store ACLs
- Deny by default
- Add rights per key
Actually running Consul
Wrap it all up

Consul

Consul is an excellent piece of software, really. I don't think I've been this excited by any other software for the last couple of years.
As they state in their Intro page : Consul has multiple components, but as a whole, it is a tool for discovering and configuring services in your infrastructure
Consul is well documented, robust, fast, replicated, datacenter aware, integrates a Key/Value store, etc... And their IRC community is very friendly.

The only major flaw I've found, is that by default, it is not secure enough.
What I mean by not secure enough, is that you must take a good care of how you configure and run the service if you don't wish to let too much opened doors to a harmful user.

Here's a list of threats I've identified - Please mind that a threat for me may very well be a nice feature for you - followed by configuration to circumvent them:

Any node could join your cluster and access sensitive information
Any user on any joined node can access cluster details/information
Any user on any joined node can execute cluster wide remote commands
Any user on any joined node can tamper with the HTTP API and
- Declare potentially harmful events
- Declare watches with potentially harmful events

Note:
All the commands and system details provided in this excerpt are with an EL6 based distribution in mind. Configurations and data paths are purely based by choice.
Please adapt accordingly :)

Binary execution rights and user running the daemon

No surprise here, this is a rule of thumbs for any daemon running on Unix/Linux, privileges must be dropped as much as possible.
Consul doesn't require you to run it as root and that's a good thing.
The thing is, that by using RPC communication, any user on the joined node can run any cluster wide command as well as remote executing command on all your cluster's nodes. It is thus important that any user cannot execute your system wide Consul binary.

Start by creating a new dedicated user without any shell access with a dedicated home directory:

useradd -d /srv/consul -s /sbin/nologin consul

On most distributions it will create a matching default group for your user as well, if not, do so.

Ensure that the home directory can only be accessed by the new consul user and its default group:

chown -R consul: /srv/consul
chmod 750 /srv/consul

Ensure the Consul binary is executable only by root user and consul group:

chown root:consul /usr/sbin/consul
chmod 750 /usr/sbin/consul

Prevent users to execute their own Consul binary

This is a bit out of the Consul scope, and more a matter of user containment, but indeed you must prevent a user to be able to upload and execute his/her own Consul binary.
The two main solutions i think of:

Mount filesystems where local users have write access with noexec flag
Trusted Path Execution from the Grsecurity patchset

Configuration access rights

It's important that you forbid any read access of your Consul configuration by other unprivileged user in order to not leak sensible settings.

mkdir /etc/consul
chown root:consul /etc/consul
chmod 750 /etc/consul

Prevent rogue nodes joining the cluster

As described in this page there's two major way to prevent that any untrusted node join your cluster

Encrypt

With the help of consul keygen command, you can generate gossip encryption key that you can use in your server/agent configuration

{
  "encrypt": "UgmMsZwR2XFNGGvJTnpHRg=="
}

This key must be used on both servers and clients for the communication to work.

TLS

TLS can/must be used to verify servers and clients authenticity. Consul requires that all servers and clients have certificates signed by a certificate authority.
You can leverage the use of your corporate PKI if you have one, just remember as per this thread that Go requires the certificate to have extendedKeyUsage clientAuth enabled.
Here's a sample configuration of enabling TLS:

Servers:

{
  "ca_file": "/etc/consul/ssl/ca_cert.pem",
  "cert_file": "/etc/consul/ssl/server.pem",
  "key_file": "/etc/consul/ssl/server.key",
  "verify_incoming": true,
  "verify_outgoing": true
}

Clients:

{
  "ca_file": "/etc/consul/ssl/ca_cert.pem",
  "cert_file": "/etc/consul/ssl/client.pem",
  "key_file": "/etc/consul/ssl/client.key",
  "verify_outgoing": true
}

Disable remote execution

Remote execution let's you execute commands and get the output of said command using
consul exec "command"
So if you were running your Consul daemon as root, running the command consul exec "shutdown -h now" would have the tremendous effect of shutting down all your Consul servers as well as all the clients nodes joined to them...
There's luckily a configuration option that should in my opinion be enabled by default:

{
  "disable_remote_exec": true
}

Now this remote execution feature could prove itself useful... This is why you limit its availability per node - Consul servers should never have it enabled - as well as its execution rights, by running the Consul binary with the unprivileged consul user we created earlier which you can couple with sudo.

Prevent HTTP API from clients

Consul HTTP API is by default available for all joined nodes on http://localhost:8500, this means that by default anyone connected on a cluster's node can use the API and/or K/V store to gather informations or change settings.

Completely

Plain and simple, use iptables to prevent access to the port:

iptables -A OUTPUT -p tcp -m tcp --dport 8500 -j REJECT

Allow some users

As you may require some of your nodes to be able to access the API and/or the K/V store, you can use the ipt_owner module of iptables:

iptables -A OUTPUT -m tcp -p tcp --dport 8500 -m owner --uid-owner consul -j ACCEPT

This would let only the consul user access the port locally, which is the user under which your daemon is running, and is unavailable to an unprivileged user.

Key/Value store ACLs

I've found the ACL system for the key/value store quite unsettling, it can be broken out like this:

Enable ACL at datacenter level
Set a master token - Not mandatory but i find it easier to manage
Chose a default policy
Set access rules to keys using API

Deny by default

As a habit, i found that denying by default is easier to manage and allow read and/or write accesses per clients.
Enabling the ACL system is a server only configuration:

{
  "acl_datacenter": "<your datacenter name>",
  "acl_master_token": "f45cbd0b-5022-47ab-8640-4eaa7c1f40f1",
  "acl_default_policy": "deny",
  "acl_down_policy": "deny"
}

acl_master_token can be easily generated with the uuidgen command but this is just to keep the same thing as Consul generates, it seems to accept any string format.

Add rights per key

You can list the ACLs using this curl command curl "http://localhost:8500/v1/acl/list?token=f45cbd0b-5022-47ab-8640-4eaa7c1f40f1&pretty=true"

As an example we'll add an ACL giving read only permission to the store key git/lastcommit.

First create a text file containing the following json code:

{                      
  "Name": "git_slave",
  "Type": "client",
  "Rules": "key git/lastcommit {policy = read}"
}

Then create the ACL using your acl_master_token:

curl -X PUT -d @kv_create_rule.json http://localhost:8500/v1/acl/create?token=f45cbd0b-5022-47ab-8640-4eaa7c1f40f1

This curl command will return you a client token that you can reuse in your agent configurations that need to access - through watches - to this key

{
  "acl_token": "d17fade7-2391-45a4-aa00-c9ed5e707e0b"
}

Actually running Consul

Here's an EL6 based init script to automatically start Consul with the correct user at startup:

#!/bin/sh
#
# consul        Start the consul daemon
#
# Author:      Olivier Mauras <olivier@mauras.ch>
#
# chkconfig: 345 99 10
# description: Starts the Consul daemon
#
# processname: consul

# Source function library.
. /etc/rc.d/init.d/functions

RETVAL=0

# Default variables
CONSUL_BIN="/usr/sbin/consul"
CONSUL_CONF="/etc/consul"
CONSUL_USER="consul"

# See how we were called.
case "$1" in
  start)
        echo -n "Starting Consul daemon: "
        daemon --check consul --user $CONSUL_USER "$CONSUL_BIN agent -config-dir=$CONSUL_CONF > /dev/null 2>&1 &"
        echo
        ;;
  stop)
        echo -n "Stopping Consul daemon: "
        killproc consul
        echo
        ;;
  status)
        status consul
        RETVAL=$?
        ;;
  restart)
        $0 stop
        $0 start
        RETVAL=$?
        ;;
  *)
        echo "Usage: consul {start|stop|status|restart}"
        exit 1
esac

exit $REVAL

Wrap it all up

I've made up an RPM package, that takes care of creating the consul user and setting up the correct rights, on the binary - It also includes the init script :). You can download the SRPM directly or just use my el6 extras repository.

Use your favorite configuration management tool - Rudder? :) - to handle the configuration deployment and /etc/consul rights.

Don't hesitate to contact me if you find inconsistencies, or if you have questions about this guide.

New package - Consul 0.4.1

2015-02-02T21:40:00+01:00

A package addition to my el6_extras repository.

Consul - 0.4.1

As usual, you can install the package using my el6_extras repository, and feel free to contact me if you encounter any issues.

New package - tinc 1.0.25

2015-01-29T12:08:00+01:00

I've been suprised to see that EPEL wasn't providing any init script for their tinc package.
Here's a version with an init script and PIE/Full RELRO support.

Tinc - 1.0.25
- Version update - from EPEL
- Provide an init script and sysconfig file
- Compile with the usual PIE and Full RELRO

One note about the init script. The goal of the script is to support multiple networks just like tinc does.
In order to do so, you have to create a symbolic link of the init script matching your network name ln -s /etc/init.d/tinc /etc/init.d/tinc.xxxx

If you need to have different options - like enabling debug - or key size on this network, copy the sysconfig file with the network name suffix cp /etc/sysconfig/tinc /etc/sysconfig/tinc.xxxx

So as usual, you can install the package using my el6_extras repository, and feel free to contact me if you encounter any issues.

Grsecurity update - 3.14.28-100

2015-01-13T20:00:00+01:00

3.14.28-100
- Update to grsecurity-3.0-3.14.28-201501120819
  Diff from the previous version on my Git repo

Sorry being slow during the holidays :)

WARNING:

At least on some servers - CPU related ? - pcid detection is still broken...

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Grsecurity update - 3.14.22-100

2014-10-31T01:10:00+01:00

3.14.22-100
- Update to grsecurity-3.0-3.14.22-201410250026
  Diff from the previous version on my Git repo

This version fixes several KVM issues

WARNING:

It appears that pcid detection is broken on latest 3.14 tree. UDEREF will make udev segfault and make the init process exit before completing boot.
The solution to fallback on slow and weak UDEREF - which in my opinion is better than having no UDEREF at all - is to pass nopcid flag to your kernel command line.
The issue has been reported to PAX team and we're waiting for a fix :)

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Qemu, libvirt and tmux updates

2014-10-24T09:40:00+02:00

I had not updated my virt repository for more than a year now... Mostly because i had not needed to.
Here's two major updates today!

Qemu - 2.1.2
- Compiled against ceph-0.80 and glusterfs-3.5.2
- Enable LZO and RDMA support
- Compiled with GCC 4.8 from devtoolset 2
Libvirt - 1.2.9
- Enable PIE, Full relro

Please mind that both qemu and libvirtd binaries require to disable PAX mprotect paxctl -cm /usr/bin/qemu* /usr/sbin/libvirtd

Tmux - 1.9a

As usual, i'm dogfooding all my packages, but it may happen that i miss a bug on an option or something, so contact me if you encounter any issues.

Grsecurity - Kernel Panic on UDP traffic

2014-10-24T09:30:00+02:00

So it appears that the issue doesn't come from the e1000 driver, but from a use-after-free in the udp protocol structure.
I've made a patch available that disables slab sanitization in udp protocol structure.
I've been toroughly testing it under heavy DNS load for two weeks and can confirm it's working well.

Indeed the drawback is that it lowers the protections on the udp protocol, that's why i won't include this patch in my default builds, just use it if you need it - I guess it will be long before we see it fixed upstream...

Grsecurity update - 3.14.21-100

2014-10-14T23:00:00+02:00

3.14.21-100
- Update to grsecurity-3.0-3.14.22-201410192047
  Diff from the previous version on my Git repo

WARNING:

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Grsecurity update - 3.14.19-101

2014-10-06T23:54:00+02:00

3.14.19-101
- Update to grsecurity-3.0-3.14.19-201409282024 Diff from the previous version on my Git repo

Note:

If you're running e1000 driver with intensive UDP traffic, you surely have encountered kernel panics.
I've reported the issue, which seems to be an upstream bug. Hope to see a patch soon.

WARNING:

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Nginx update - 1.62-2

2014-09-26T00:45:00+02:00

Nginx - 1.62
- Compile with kernel 4.8 from devtoolset2 to enable PIE
- Remove Perl RPATH

As usual, i'm dogfooding all my packages, but it may happen that i miss a bug on an option or something, so contact me if you encounter any issues.

Exim update - 4.84-1

2014-09-20T15:55:00+02:00

Exim - 4.84
- latest stable version

As usual, i'm dogfooding all my packages, but it may happen that i miss a bug on an option or something, so contact me if you encounter any issues …

Grsecurity update - 3.14.19-100

2014-09-18T23:50:00+02:00

3.14.18-100
- Update to grsecurity-3.0-3.14.18-201409141906
  Diff from the previous version on my Git repo
  I just had compiled this package when 3.14.19 release got out, so this version is untested
3.14.19-100
- Update to grsecurity-3.0-3.14.19-201409180900
  Diff from the previous version on my Git repo

I've jumped the packaging of several versions since 3.14.17-100 but as usual, all the branches are available in my Git repo

Note:

If you're running e1000 driver with intensive UDP traffic, you surely have encountered kernel panics.
I've reported the issue, which seems to be an upstream bug. Hope to see a patch soon.

WARNING:

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

EL6 Extras updates and new packages

2014-09-18T23:30:00+02:00

A bunch of new packages/versions:

Nginx - 1.6.2
- latest stable version fixing CVE-2014-3616
Uwsgi - 2.0.6
- New modular build with multiple language support
  - Lua 5.1 - lua plugin
  - Python 2.6 - python plugin
  - Python 2.7 - python27 plugin
  - Python 2.6 - python33 plugin
  - PHP 5.3 - php plugin
  - PHP 5.4 - php54 plugin please see below
  - CGI is now a plugin as well
    Python 2.7 and 3.3 requires that you enable the software collections repository
    Next version will contain init script and perl support
PHP 5.4 - 5.4.16-116
- Include mcrypt module
- Include -embedded package
  I was kinda pissed off by RH purposely removing the support of PHP embedded library in their software collections versions, as well as still not supporting mcrypt module, so i just rebuilded it with a higher release number (+100).
  You still need to enable software collections repository for php54-php-pear* dependancies.
Libmcrypt - 2.5.8
Basically a rebuild of PUIAS SRPM

As usual, i'm dogfooding all my packages, but it may happen that i miss a bug on an option or something, so contact me if you encounter any issues.

Grsecurity update - 3.14.17-100

2014-08-22T00:00:00+02:00

3.14.17-100
- Update to grsecurity-3.0-3.14.17-201408212334
  Diff from the previous version on my Git repo

I've jumped the packaging of several versions since 3.14.15-102 but all the branches are available in my Git repo

WARNING:

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Grsecurity update - 3.14.15-102

2014-08-05T19:41:00+02:00

3.14.15-102
- Update to grsecurity-3.0-3.14.15-201408032014
  Diff from the previous version on my Git repo

WARNING:

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Grsecurity update - 3.14.15-101

2014-08-01T22:40:00+02:00

3.14.15-101
- Update to grsecurity-3.0-3.14.15-201408010644
  Diff from the previous version on my Git repo

Even if i haven't built the last two Grsecurity versions, their branches are up in my Git repo

WARNING:

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Grsecurity update - 3.14.13-100

2014-07-24T06:00:00+02:00

3.14.12-102
- Update to grsecurity-3.0-3.14.12-201407170638
  Diff from the previous version on my Git repo
3.14.13-100
- Update to grsecurity-3.0-3.14.13-201407232159
  Diff from the previous version on my Git repo

WARNING:

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Gradm update

2014-07-17T04:55:00+02:00

3.0-201405281853_1
- Version bump

Complete changelog can be found here

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Grsecurity update - 3.14.12-101

2014-07-16T19:00:00+02:00

3.14.12-101
- Update to grsecurity-3.0-3.14.12-201407151838

You can see a diff from the previous version on my Git repo

WARNING:

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Grsecurity update - 3.14.12-100

2014-07-10T13:00:00+02:00

3.14.12-100
- Update to grsecurity-3.0-3.14.12-201407100035

You can see a diff from the previous version on my Git repo

WARNING:

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Grsecurity update - 3.14.11-101

2014-07-09T16:10:00+02:00

3.14.11-101
- Update grsecurity-3.0-3.14.11-201407081919

You can see a diff from the previous version on my Git repo

WARNING:

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Grsecurity update - 3.14.10-101

2014-07-06T15:01:00+02:00

3.14.10-101
- Update grsecurity-3.0-3.14.10-201407052031

You can see a diff from the previous version on my Git repo

WARNING:

# dmesg | grep UDEREF 
PAX: slow and weak UDEREF enabled

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Nginx update to new stable

2014-07-05T20:49:00+02:00

Nginx latest stable version - 1.6.0

As usual, contact me if you encounter any issues.

Grsecurity update - 3.14.10

2014-07-02T20:00:00+02:00

And there's another one today :)

3.14.10-100
- Update grsecurity-3.14.10-201407012152

You can see a diff from the previous version on my Git repo

As usual RPMs are available using the YUM repository, and don't hesitate to contact me if you find any issue.

Grsecurity update - 3.14.9

2014-06-27T20:00:00+02:00

And there's another one today :)

3.14.9-100
- Update grsecurity-3.0-3.14.9-201406262057

You surely have noted that the two latest kernels come only in nozfs flavor. This is indeed correct, and that's because i'm wondering if it makes any sense to continue supporting both at the same time... I'm not so sure about it.

As usual RPMs are available using the YUM repository, and don't hsitate to contact me if you find any issue.

Grsecurity update to new stable kernel

2014-06-26T21:00:00+02:00

It's here!! Grsecurity finally chose Kernel 3.14 as the new stable kernel.
I know i could have updated the 3.2 branch while waiting, but i secretely hopped for the decision to come sooner ;)

Here's the changelog compared to my latest 3.2 kernel.

3.14.8-100
- Update to devtoolset 2 with GCC 4.8
- Don't apply grsecurity patch anymore use http://git.mauras.ch/linux-grsecurity/ tagged archives
- Enable CONFIG_PAX_RANDKSTACK and CONFIG_GRKERNSEC_KSTACKOVERFLOW
- Force dracut's detection of virtio_blk module - see bug

Yes, aside the version, the biggest change is the way i will now handle kernel archives.
I'll now track all the Grsecurity patches in separate branches in my Git repo. This makes packaging easier, aswell as comparing patch versions.
I'm still thinking of doing a repo containing all my RPM spec files to make your life easier, just have to find the correct way to have something nice and well sorted.

As usual RPMs are available using the YUM repository, and don't hsitate to contact me if you find any issue.

Nginx + Uwsgi + Cgit

2014-06-25T01:36:00+02:00

As you've seen in my previous post, there's been some package updates in the EL6 Extras repository, so let's have some fun with them!

First you indeed need my EL6 Extras repository if it's not done yet, please start with the repositories page.

Install the packages

Now that you're good with the repository, install the packages:

# yum -y install nginx uwsgi cgit

Configure Cgit

The default config from the package is pretty sane, the only thing you'll need to add to your /etc/gitrc - And indeed your Git repositories config - is:

virtual-root=/

This will keep cgit from adding cgit.cgi to your URL.
To enable syntax highlighting, uncomment the following line in your /etc/cgitrc

source-filter=/usr/libexec/cgit/filters/syntax-highlighting.py

Then install pygments package

# yum -y install python-pygments

Configure Uwsgi

Just add this simple conf

# mkdir /etc/uwsgi
# cat << EOF > /etc/uwsgi/cgit.ini
[uwsgi]
socket = 127.0.0.1:8080
uid = cgit
gid = cgit
processes = 1
threads = 2
cgi = /usr/libexec/cgit/cgit.cgi
EOF

There's no init script provided in my package yet, so just manually run - or any way you want

# uwsgi --ini /etc/uwsgi/cgit.ini &

Configure Nginx

Finally just add this simple server block to your Nginx config and customize it to your needs

server {
    # Change this to your taste
    listen       80 default_server;
    server_name  _;

    # We want to serve cgit static content from /usr/share/cgit and let Nginx cache it
    location ~* ^.+(cgit.(css|png)|favicon.ico) {
        root /usr/share/cgit;
        expires 30d;
    }

    # Now forward everything else to our uwsgi instance
    location / {
        include uwsgi_params;
        uwsgi_modifier1 9;
        uwsgi_pass 127.0.0.1:8080;
    }
}

Start Nginx, and enjoy your Cgit running on your server_name

As usual, contact me if you have any question.

EL6 Extras updates

2014-06-25T01:20:00+02:00

I've been quite slow to keep up with updates for the last 3 months.
Here's a bunch of updates:

Nginx - Update to latest stable release 1.4.7
Uwsgi
- Update to latest release 2.0.5.1
- Now also support CGI plugin
Cgit
- Update to latest release 0.10.1
- Major refactoring of the package
  - Add /var/cache/cgit directory
  - Fix syntax-highlight.py
  - Modify cgitrc for a saner default config
  - Create cgit user and group
  - Overral files/paths cleanup

As usual, contact me if you encounter any issues.

Rudder: Fun with variables

2014-06-19T02:41:00+02:00

How many different configuration files do you maintain for the same service spread over your different Rudder groups?
Personnaly only one!

I hate having to maintain multiple - almost - identical configuration files, it's counterproductive and definitely prone to errors. Having to change one setting in twenty files is always a painful task.
Hopefully Rudder let's you easily manage your config file by keeping the relevant parts of it as variables and i'll show you how to get the most out of it.

Modifying default technique

I started this post so long ago that this point is not even needed anymore. This commit merged it so you should have it by default on any latest Rudder minor version - 2.10.2, 2.9.5 and 2.6.14.

While, in this example, we're going to mainly use genericVariableDefinition and checkGenericFileContent techniques, since Rudder 2.7, you can also set parameters and reuse them as variables inside directives.

Using variables to dynamically change settings in enforced files

sshd_config seems like a good example, i use the same default configuration - applied on all nodes - everywhere with only slight changes in AllowGroups section depending in which group the managed node is located.

Without using variables, i'd have to use at least 10 different directives to apply to my node groups, with actually only a tiny part of a line different from the default...
If for some reason i'd want to change a default setting - not related to my AllowGroups section, let's say PermitRootLogin - i'd then have to modify 10 directives, and this would be painful...
Modifying a directive is at least 3 manual actions + 1 rule generation, so i would end up with 30 manual actions + 10 rule generations.
Not really efficient is it ? And along the way I could just mess one of the file syntax, rendering sshd unavailable for some of my nodes.

How does it look like?

First what do we want to achieve? We want that our sshd_config AllowGroups always contain the group admins and that some node groups also contain the group not_admins.

Let's start by creating a default_admin_group parameter - It's the default you should find everywhere on all your nodes, it makes sense to have it as a parameter.
click on images to get them in fullscreen

Now create a new Generic CFEngine variable definition directive with the default priority - 5.
This one will be applied to all your nodes.

Create a second one with priority set to 9 this time - Actually anything higher than 5.
This one will be applied only to the node groups where you want the group not_admins The higher priority means that this directive will be applied after the first one and consequently override ${generic_variable_definition.SSH_ALLOWED}.

Finally create your default Enforce a file content directive.

In plain text

${rudder.rudder_file_edit_header}
# stripped down default sshd config for the sake of example
LoginGraceTime 30s
PermitRootLogin no
StrictModes yes
MaxAuthTries 3
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
X11Forwarding yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem   sftp    /usr/libexec/openssh/sftp-server
AllowGroups ${generic_variable_definition.SSH_ALLOWED}

Once applied on your nodes you'll get this

#########################################
### Managed by Rudder, edit with care ###
#########################################
# stripped down default sshd config for the sake of example
LoginGraceTime 30s
PermitRootLogin no
StrictModes yes
MaxAuthTries 3
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
X11Forwarding yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem   sftp    /usr/libexec/openssh/sftp-server
AllowGroups admins

Or depending if you apply the second generic variable directive with the priority 9

#########################################
### Managed by Rudder, edit with care ###
#########################################
# stripped down default sshd config for the sake of example
LoginGraceTime 30s
PermitRootLogin no
StrictModes yes
MaxAuthTries 3
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
X11Forwarding yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem   sftp    /usr/libexec/openssh/sftp-server
AllowGroups admins not_admins

Now you can just easily add/change settings of your sshd service in a single location.

Even more fun?

You can also set empty values for your variables in the Generic CFEngine variable definition, this will let you insert default empty values replaced by actual content when needed.

Default variable

variable name: SFTP
variable content:

You set an empty variable here because you need your default configuration to have nothing set. In this example only your SFTP nodes group will have something configured.

For your SFTP nodes group

variable name: SFTP
variable content: Match group sftpusers$(const.n)$(const.t)ChrootDirectory /sftp/$(const.n)$(const.t)ForceCommand internal-sftp

Your default Enforce a file content directive

${rudder.rudder_file_edit_header}
# stripped down default sshd config for the sake of example
LoginGraceTime 30s
PermitRootLogin no
StrictModes yes
MaxAuthTries 3
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
X11Forwarding yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem   sftp    /usr/libexec/openssh/sftp-server
AllowGroups ${generic_variable_definition.SSH_ALLOWED}

# Only populated on SFTP servers
${generic_variable_definition.SFTP}

Which would be expanded to

#########################################
### Managed by Rudder, edit with care ###
#########################################
# stripped down default sshd config for the sake of example
LoginGraceTime 30s
PermitRootLogin no
StrictModes yes
MaxAuthTries 3
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
X11Forwarding yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem   sftp    /usr/libexec/openssh/sftp-server
AllowGroups admins not_admins

# Only populated on SFTP servers
Match group sftpusers
    ChrootDirectory /sftp/
    ForceCommand internal-sftp

Hope you'll find this article useful and as usual, feel free to contact me if you have any question!

Grsecurity updates and kernel-test "EOL"

2014-04-05T00:00:00+02:00

New versions of my grsecurity kernels.

3.2.55-4 and 3.2.55-104
- Upgrade to grsecurity-3.0-3.2.55-201403252026.patch
- Set CONFIG_GRKERNSEC_FLOODBURST=1000

Yes they were actually available in the repo since end of march but i hadn't time to blog about it.
Please also note that i'm not gonna update kernel-test anymore. It takes too much time to keep up, and Grsecurity stable kernel should soon switch to 3.13 or 3.14 - Depending what Canonical decides for their 14.04 LTS.
I'll let the reposiroty as it is for those interested in SRPMs.

As usual RPMs are available using the YUM repository

Don't hesitate to contact me if you find any issue.

Unbound update - Now with pythonmodule!

2014-03-22T12:45:00+01:00

Unbound latest version - 1.4.22
Notable differences:

Doesn't depend on ldns anymore - yum -y update unbound && yum -y remove ldns
Built with pythonmodule support

Init script will populate chroot with your python2.6 directory so that python scripts can be fully usable.

As usual, contact me if you encounter any issues.

Nginx update

2014-03-07T17:00:00+01:00

Nginx latest stable version - 1.4.6

As usual, contact me if you encounter any issues.

EL6 Extras update

2014-03-01T15:47:00+01:00

Keeping up with non up to date packages. Please find latest version 2.68 of dnsmasq

As usual, contact me if you encounter any issues.

EL6 Extras additions and Grsecurity kernels

2014-02-26T20:00:00+01:00

El6 Extras

Updates:
- Nginx 1.4.5
Additions:
- Exim 4.82
- Libgsasl 1.8.0 - needed by exim
- Unbound 1.4.21
- Ldns 1.6.17 - needed by unbound

The usual, Full RELRO, PIE, no RPATH and up to date builds.

Grsecurity

Completely re-arranged the repository, please update your repo file
I added a new kernel without ZFS compatibility. It will be numbered starting 100 to differenciate from the classic stable with ZFS kernel. Please enable kernel-nozfs repo in your grsecurity.repo file.
Updates:
- Kernel stable 3.2.55 to grsecurity-3.0-3.2.55-201402152203.patch
  - kernel-nozfs has the RANDSTRUCT option enabled
- ZFS modules now synced from the git master repo of zfsonlinux - Still have snapshot automount issue.

As usual please contact me if you encounter any issues with these packages.

EL6 Extras package additions

2014-01-31T07:00:00+01:00

I forgot to port my old tmux package over the new EL6 Extras repo, so here comes the addition.

Please note that i updated it to version 1.8 which now requires the use of libevent 2 - since version 1.7 actually, tmux 1.6 was the last version compatible with libevent 1.4 that you may find on EPEL.
To keep compatibility between libevent versions, libevent2 package will install itself in /opt/el6_extras.
Starting from now, i'll use this directory to support multiple versions of apps/libs.

As usual you'll have to use the GPG public key to verify packages signature.

Grsecurity updates

2014-01-24T00:00:00+01:00

I've been quite busy during December with holidays and all, so here's my attempt to keep up with up to date grsecurity patches.

Stable 3.2.54 kernel
Test 3.12.8 kernel
Gradm 201401231926
spl/zfs Ryao's gentoo-next branch

Ryao is one of the main zfsonlinux developpers which is also maintaining Gentoo's ZFS port with hardened profile in mind. Switching to Ryao's gentoo-next branch thus makes more sense as Ryao tightly follows Grsecurity changes that could break ZFS.
These new ZFS packages are fixing the failed user helper error introduced by the new restrictions in Grsecurity 3.0. It still doesn't fix the ZFS snapshot automount functionnality though, but i did report it to Ryao and hope to be able to quickly provide a fixed package.

As usual RPMs are available using the YUM repository

Changelog: Grsecurity changelogs can be found here: stable and test
spl/zfs gentoo-next diff are there: spl and zfs

Don't hesitate to contact me if you find any issue.

Rudder agent configuration at node first boot

2014-01-24T00:00:00+01:00

After migrating all my rules from Cfengine to Rudder, i was still missing something. All my freshly kickstarted nodes weren't by default integrated to Rudder.
While it was fairly easy on my Cfengine installation to fetch rules for a new node during the kickstart phase, the fact that Rudder requires the node to be accepted first, was kind of breaking my server provisioning workflow.

Hopefully Rudder API provides everything's needed to programatically accept a new node on your Rudder server.
The following init script is to be used at your node's first boot. It will send its inventory to the server, accept it from the pending nodes list and then fetch and apply the rules.

I consider that you've been able to install rudder-agent and activate this script from you kickstart install prior to the node's first boot.

rudder-installer init script:

#!/bin/bash
## BEGIN INIT INFO
# Provides: rudder-installer
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 3 4 6
# Required-Start:
#
## END INIT INFO
# rudder-installer: Configure rudder agent
# chkconfig: 345 70 99
# description: Configure rudder agent

# Source function library.
. /etc/init.d/functions

# Variables
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin
API_TOKEN="<YOUR API TOKEN>"
RUDDER_SRV="<YOUR RUDDER SERVER URL>"
RUDDER_SRV_HOSTNAME="<YOUR RUDDER SERVER HOSTNAME>"

# This will remove the script if configuration ended up correctly
clean() {
    echo -n "Removing starting script"
    chkconfig --del rudder-installer
    rm -f /etc/init.d/rudder-installer &> /dev/null
    RETVAL=$?
    return $RETVAL
}

# Configuration of agent
start() {
    echo "Configuring rudder agent"
    # Configure server
    echo $RUDDER_SRV_HOSTNAME > /var/rudder/cfengine-community/policy_server.dat

    # Now declare host on server
    echo "    - Sending node inventory to the server - Can take up to 10mn please be patient..."
    /opt/rudder/bin/cf-agent -KI -D force_inventory &>> /root/rudder.log

    # My inotify script will process the new inventory in far less than 10s, but better safe than sorry
    echo "    - Actually sleeps 10s as it's the needed time to process inventory"
    sleep 10

    echo "    - Verify that node is now pending on the server"
    # Ask server using API
    I=0
    RESULT=$(curl -k -X GET -H 'Content-Type: application/json' -H "X-API-Token: ${API_TOKEN}" -H "X-API-Version: 2" ${RUDDER_SRV}/rudder/api/nodes/pending 2>> /root/rudder.log | sed 's/,/\n/g' | grep $HOSTNAME -B 2)
    RET=$?

    # Looping to check if node is in pending list
    while [ $RET != 0 ]; do
        sleep 1
        I=$((${I}+1))
        if [ $I -lt 30 ]; then
            RESULT=$(curl -k -X GET -H 'Content-Type: application/json' -H "X-API-Token: ${API_TOKEN}" -H "X-API-Version: 2" ${RUDDER_SRV}/rudder/api/nodes/pending 2>> /root/rudder.log | sed 's/,/\n/g' | grep $HOSTNAME -B 2)
            RET=$?
        else
            # We exit if the node is still not in pending list
            echo "!! Inventory discovery took too long, something must have fail - Exiting"
            sleep 5
            exit
        fi
    done

    echo "    - Get the node ID from the server"
    NODEID=$(echo $RESULT | sed 's/.*id\":\"//;s/\".*//')
    RET=$?

    echo "    - Auto accept node"
    echo "Accepting node" >> /root/rudder.log
    curl -k -X POST -H "X-API-Version: latest" -H "X-API-Token: ${API_TOKEN}" ${RUDDER_SRV}/rudder/api/nodes/pending/${NODEID} -d "status=accepted" &>> /root/rudder.log

    # At least should have :)
    echo "    - Node has been accepted!"

    # Yes 6mn is a safe waiting time on my server which takes ~4mn30 to process new server rules
    echo "    - Now that the node is accepted wait for 6 mn while Rudder server rebuilds rules"
    i=0
    # Here's the ugly "Make the user wait" timer
    while [ $i -lt 360 ]; do
        i=$(($i+1))
        sleep 1
        if [ $i == 30 ]; then
            echo -ne "$i"
        elif [ $i == 60 ]; then
            echo -ne "$i"
        elif [ $i == 90 ]; then
            echo -ne "$i"
        elif [ $i == 120 ]; then
            echo -ne "$i"
        elif [ $i == 150 ]; then
            echo -ne "$i"
        elif [ $i == 180 ]; then
            echo -ne "$i"
        elif [ $i == 210 ]; then
            echo -ne "$i"
        elif [ $i == 240 ]; then
            echo -ne "$i"
        elif [ $i == 270 ]; then
            echo -ne "$i"
        elif [ $i == 300 ]; then
            echo -ne "$i"
        elif [ $i == 330 ]; then
            echo -ne "$i"
        else
            echo -ne "."
        fi
    done

    # Ok rules should be generated now let's apply them on the node
    # I run two first command twice to ensure everything's good - once should be sufficient
    echo -e "\n    - Updating rules"
    echo "### Ensure UUID is correctly detected by the node ###" >> /root/rudder.log
    /opt/rudder/bin/cf-agent -KI &>> /root/rudder.log
    /opt/rudder/bin/cf-agent -KI &>> /root/rudder.log
    echo "### Update rules ###" >> /root/rudder.log
    /opt/rudder/bin/cf-agent -KI -b update &>> /root/rudder.log
    /opt/rudder/bin/cf-agent -KI -b update &>> /root/rudder.log
    echo "### Now apply them ###" >> /root/rudder.log
    /opt/rudder/bin/cf-agent -KI &>> /root/rudder.log

    RETVAL=$?
    return $RETVAL
}

# Init script parameters stuff
case "$1" in
    start)
        start && success || failure
        [[ $? == 0 ]] && clean && success || failure
        echo
        ;;
    *)
        echo $"Usage: $0 start"
        exit 3
        ;;
esac

So upon completion, the script will remove itself from the init sequence and your freshly installed node should have all required rules applied. If not, consult /root/rudder.log for details.
You have to use my inotify script on the server, as if you don't you will encounter the infamous "5mn delay" before your node appears in the Rudder new nodes pending list which would render this init script completely useless as it would exit without applying any rules.

As usual, don't hesitate to contact me if you have any question.

What's using swap

2014-01-14T21:00:00+01:00

If like me you're bored of looping over all /proc/<pid>/status files to find which application is actually using swap, here a tiny python script to ease the task.

The script outputs a list of processes using swap with the following format: <name> <uid> <swap usage>

Here's an example of what you can expect of the output of vmswap.py | sort -nk3:

pampd 501 4060kB
spampd 501 4100kB
spampd 501 4228kB
rsyslogd 0 4328kB
spampd 501 4368kB
spampd 501 7576kB
spampd 501 7612kB
httpd 48 10100kB
httpd 48 10140kB
httpd 48 10188kB
httpd 48 10216kB
httpd 48 10284kB
httpd 48 10308kB
httpd 48 10332kB
httpd 48 10388kB
httpd 48 10616kB
httpd 48 10628kB
httpd 48 10636kB
httpd 48 10848kB
httpd 48 10908kB
httpd 48 10916kB
httpd 48 10972kB
httpd 48 10980kB
httpd 48 11000kB
httpd 0 11084kB
httpd 48 11276kB
httpd 48 13056kB
httpd 48 14076kB
miniserv.pl 0 17412kB
memcached 496 23796kB
named 25 42388kB
mysqld 27 46292kB

And here's vmswap.py:

#!/bin/env python2
import os
import types
import subprocess

def get_command_output(cmd):
    # Get its output 
    process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=None, shell=False)
    output = process.communicate()
    return output[0].split('\n')

def main():
    # Defaults ps command and arguments
    ps_cmd = 'ps'
    ps_args = 'axcu'

    # Build ps command
    ps_cmd = '%s %s' % (ps_cmd, ps_args)

    # Get command output
    plist = get_command_output(ps_cmd)

    # Declare final list
    flist = []
    fadd  = flist.append
    proc = {}

    # Loop on output list
    for pline in plist:
        # Make a list out of each line
        proc_list = pline.split()
        if proc_list and os.path.isfile(os.path.join('/proc', proc_list[1], 'status')):
            proc[proc_list[1]] = {}
            for line in open(os.path.join('/proc', proc_list[1], 'status'), 'rb'):
                if 'Name:' in line:
                    pname = line.strip('\n')
                    pname = pname.split('\t')[1]
                    proc[proc_list[1]]['name'] = pname
                elif 'Uid:' in line:
                    puid = line.split('\t')[1]
                    proc[proc_list[1]]['uid'] = puid
                elif 'VmSwap:' in line:
                    pswap = line.strip('\n')
                    pswap = pswap.split('\t')[1]
                    proc[proc_list[1]]['swap'] = " ".join(pswap.split())

    for pid in proc:
        if 'swap' in proc[pid]:
            if proc[pid]['swap'] != '0 kB':
                output = '%s %s %s' % (proc[pid]['name'], proc[pid]['uid'], proc[pid]['swap'].replace(" ", ""))
                print(output)

if __name__ == '__main__':
    main()

Don't hesitate to contact me if you have any question or suggestion.

Grsecurity kernels update

2013-12-02T00:00:00+01:00

Second grsecurity patch in less than a week :)

Stable 3.2.53 kernel
Test 3.12.2 kernel

WARNING!

Please don't use 3.2.52 package. It was built without important virtio drivers which would make it kernel panic under a KVM guest.
Please also be aware that this Grsecurity patch enforcing kernel helpers in /sbin will prevent zfs automounting snapshots.

As usual RPMs are available using the YUM repository

Changelog: Grsecurity changelogs can be found here: stable and test

Don't hesitate to contact me if you find any issue.

Grsecurity update to latest 3.0 patch

2013-11-29T00:00:00+01:00

Here comes the new packages supporting the new Grsecurity patch version!
This mean:

Stable 3.2.52 kernel
- And its 0.6.2 ZFS modules
Test 3.12.1 kernel
- ZFS modules has been updated latest master git branch for 3.12 compatibility
Gradm 3.0

You can note the return - if you ever noticed it was gone - of the stable kernel. Finally had time to catch up with the stable releases and add the configuration improvement made on the test kernels.

WARNING!

Please note that size overflow plugin has been modified to have more coverage. This could mean that something working on a previous kernel could be broken on this new one - modules compilation etc...
It seems to impact only the stable 3.2.52 kernel though but USE AT YOUR OWN RISK!

As usual RPMs are available using the YUM repository

Changelog:

Kernel 3.12.1:
- Update to grsecurity-3.0-3.12.1-201311261522.patch
- Add option CONFIG_GRKERNSEC_JIT_HARDEN
- Add option CONFIG_GRKERNSEC_HARDEN_IPC

Kernel 3.2.52:
- Upgrade to grsecurity-3.0-3.2.52-201311261520.patch
- Add all options added to test
- Remove overlayfs patch
- Add option CONFIG_GRKERNSEC_JIT_HARDEN

ZFS/SPL modules - kernel-test only:
- Update to git master branch - 16_g30607d9

Gradm:
- Update to 3.0
- Remove management of /dev/grsec through the package

Don't hesitate to contact me if you find any issue.

Nginx package update

2013-11-19T20:25:00+01:00

I guess you've heard about CVE-2013-4547, so here's version 1.4.4 of Nginx.
As usual it's available directly from my EL6 extras repository and you'll have to use the GPG public key to verify packages signature.

Easy git access control - Part 2 - more granularity

2013-11-11T21:00:00+01:00

In my previous post i talked about how to easily manage write permissions on a git repository.
As i said, the script could easily be extented to provide more granularity.
Here comes an improved version!

Features:

Support groups
Support nested groups
Let's you name the admin group
Let's you give full rights to admin group
Let's you give user the right to push to his/her own branch - branch named after user name

Here's an example git.acl:

[global]
# Name of the admin group
admin_group     = admin
# Is admin group able to push everything? 0|1
full_admin      = 1
# Is user able to push in his named branch? 0|1
owner_branch    = 1

[groups]
admin       = olivier joe
commiters   = @admin jane jim
project1    = @admin @commiters @project2
project2    = @admin john mickael

[branches]
master      = @admin
project1    = @project1
test_branch = sam

And here is the updated update hook:

#!/bin/env python2
import os
import sys
import subprocess
import ConfigParser
#from pprint import pprint

# Exit on error
def error_exit(string):
    print string
    sys.exit(1)

# Exit on success
def ok_exit(string):
    print string
    sys.exit(0)

# Recursively get all users from a group
def expand_nested_groups(groups, group, user, parsed_groups=[]):
    parsed_groups.append(group)
    if user.startswith('@'):
        group_name = user[1:]
        if group_name in parsed_groups:
            return []
        res = set()
        for u in groups[group_name]:
            res.update(expand_nested_groups(
                groups,
                group_name,
                u,
                parsed_groups))
        return res
    return [user]

# Verify that acl_file exists before reading it
def _config():
    # Fix acl file path if needed
    exec_path = os.getcwd()
    if '/hooks/' in exec_path:
        acl_file = './git.acl'
    else:
        acl_file = './hooks/git.acl'

    # Open and read acl file if exists
    if os.path.isfile(acl_file):
        config = ConfigParser.RawConfigParser()
        if not config.read(acl_file):
            error_exit(acl_file+" doesn't seem to be a valid config file!")
        return config
    else:
        error_exit(acl_file+" is missing!")

# Returns global settings 
def _config_global(config, option, is_int):
    # Verify if global section exists
    if config.has_section('global'):
        if config.has_option('global', option):
            if is_int:
                return config.getint('global', option)
            else:
                return config.get('global', option)
        else:
            error_exit(str(option)+" option doesn't exists in 'global' section!")
    else:
        error_exit("'[global]' section not found in your acl file!")

# Retuns a dict of groups
def _config_groups(config):
    if config.has_section('groups'):
        group_list = config.items('groups')
        if group_list:
            group_dict = dict((e[0], e[1].split()) for e in group_list)
            #pprint(group_dict)

            # Let's expand groups
            expanded_groups = {}
            for group, users in group_dict.iteritems():
                expanded_groups[group] = set()
                for user in users:
                    expanded_users = expand_nested_groups(
                        group_dict,
                        group,
                        user,
                        [])
                    expanded_groups[group].update(expanded_users)

            return expanded_groups
        else:
            error_exit("Your '[groups]' section seems to have no options...")
    else:
        error_exit("'[groups]' section not found in your acl file!")

# Returns a list of users allowed for a branch
def _config_branch(config, branch):
    if config.has_section('branches'):
        if config.has_option('branches', branch):
            return config.get('branches', branch).split()
        else:
            error_exit("This branch doesn't have any right set!")
    else:
        error_exit("'[branches]' section not found in your acl file!")

def main():
    # Get arguments
    refname = sys.argv[1]
    oldrev = sys.argv[2]
    newrev = sys.argv[3]
    gituser = os.getenv('USER')

    # Declare object types
    objtype = ['commit', 'delete', 'tag']

    # Init acl file reading
    acl_config = _config()

    # Get global settings
    admin_group = _config_global(acl_config, 'admin_group', 0)
    full_admin = _config_global(acl_config, 'full_admin', 1)
    owner_branch = _config_global(acl_config, 'owner_branch', 1)

    # Get group list
    g = _config_groups(acl_config)
    #pprint(g)

    # Get branch name
    exp_refname = refname.split('/')
    branch_name = exp_refname[2]

    # Printing this only for information
    print "branch="+branch_name
    print "oldrev="+oldrev
    print "newrev="+newrev
    print "user="+gituser

    # Get object type
    cmd = 'git cat-file -t %s' % (newrev)
    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=None, shell=True)
    output = process.communicate()
    thistype = output[0][:-1]
    print "object="+thistype

    # Reject unknown object types 
    if thistype not in objtype:
        error_exit("You can't push this type of objects")

    ### Access right checking ###
    # Admin processing
    if full_admin and gituser in g[admin_group]:
        ok_exit('ADMIN ACCESS GRANTED')

    # user writes to his own branch
    if owner_branch and gituser == branch_name:    
        ok_exit('BRANCH OWNER ACCESS GRANTED')

    # Not admin nor branch owner
    # Get branch access rights
    branch_access = _config_branch(acl_config, branch_name)

    # Branch exists in git.acl check if user has rights
    # Start with group
    for right in branch_access:
        if '@' in right and gituser in g[right.strip('@')]:
            ok_exit('BRANCH ACCESS GRANTED')

    # Not in a group so do a basic check
    if gituser in branch_access:
        ok_exit('BRANCH ACCESS GRANTED')
    else:
        error_exit('BRANCH ACCESS DENIED')

    # Catchall exit, means we haven't match any case...
    sys.exit(1)


if __name__ == '__main__':
    main()

Thanks to Steeve Chailloux for his group expansion function which was way better than mine :)

Next time let's support rights on tags ;)

Package addition in EL6 repo

2013-10-25T15:00:00+02:00

I've added some new packages in the EL6 extras repository

GeoIP 1.5.1
nginx 1.4.3
libyaml 0.1.4
PyYAML 3.10
uwsgi 1.9.18.2

As usual please use the GPG public key to verify packages signatures.

Easy git access control

2013-10-21T09:25:00+02:00

Basic access to a git repository is fairly easy.
You specify --shared when initializing your bare repository, chgrp <your group> to the repository directory and your done. Now everyone in <your group> has read and write access to the repository with their account.

So what do I need?

An easy way to tell which user can actually write or not
A solution easy to integrate with my actual configuration manager - Here rudder/cfengine

Gitolite - to name the widely used tool for this kind of case - requires to install a bunch of script files, set a common git user, and store all its settings in a specific git repository. Definitely not practical to integrate in a configuration management tool.

How to do it then?

It is actually very easy to achieve these requirements with only basic linux settings and git hooks.

Read access:
- ssh + user group
Write access:
- git hook + acl file

Create your new git repository:

git init --bare --shared <repo>
chgrp -R <your group> <repo>
chmod o-rx <repo>
ls -ld <repo>
drwxrws--- 7 root <your group> 10 Oct 21 09:40 <repo>

Replace <repo>/hooks/update by the following python script.
This script checks the type of command you're sending and validates if your username is listed in the git.acl file and exits accordingly.

#!/bin/env python
import os, sys, subprocess, ConfigParser

refname = sys.argv[1]
oldrev = sys.argv[2]
newrev = sys.argv[3]
user = os.getenv('USER')

revtype = ['commit', 'delete', 'tag']

config = ConfigParser.RawConfigParser()
config.read('./git.acl')
users = config.get('users', 'write').split()


print refname
print "oldrev="+oldrev
print "newrev="+newrev
print user

cmd = 'git cat-file -t %s' % (newrev)
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=None, shell=True)
output = process.communicate()
thistype = output[0][:-1]

if ( thistype in revtype and not user in users):
    print "!!! You're not authorized to push code to this repo"
    sys.exit(1)

sys.exit(0)

Here is <repo>/hooks/git.acl

[users]
write = admin joe bob brad

The script could easily be extented to do more granular rights - allow pushing to branches but not to master, which branches a user could push to, etc... - while still being very easy to deploy using your favorite configuration management tool.

Grsecurity 3.11.6 kernel available

2013-10-19T09:20:00+02:00

Grsecurity test kernel is now available as RPM in the repository
You can fetch the SRPM here

Changelog:

- Update to kernel 3.11.6
- Update to grsecurity-2.9.1-3.11.6-201310191259.patch
- Add new grsec option: CONFIG_GRKERNSEC_DENYUSB
- overlayfs patch is removed

At the same time, zfs modules have been recompiled for it.
You can also install them by using the yum repository.

Please note that now all packages are signed using GPG. You can import the signing GPG public key from here
Until i create RPM packages to deploy my repository configs, you can import the key using: rpm --import http://repos.mauras.ch/RPM-GPG-KEY-build_at_mauras.ch

Rudder: Migrate from cfengine

2013-09-10T00:00:00+02:00

The following standalone promise installs rudder-agent on a currently managed cfengine host.
The container class is here only for me to identify OpenVZ guests.

Use at your own risk and modify it at your convenience.

bundle agent rudder_agent_install {
    vars:
        container::
            "repo_content" string => "[Rudder]
name=Rudder 2.7 Repository
baseurl=http://repos/rudder/test/RHEL_6/
gpgcheck=0
";

    files:
        container::
            "/etc/yum.repos.d/rudder.repo"
                create => "true",
                edit_line => append_to_repo_file("$(repo_content)");

    packages:
        container::
            "rudder-agent"
                package_policy => "add",
                package_method => yum_rpm_sta,
                classes => repaired_("kick_inventory");

    methods:
        kick_inventory::
            "any" usebundle => rudder_config;

}

bundle agent rudder_config {
    files:
        "/var/rudder/cfengine-community/policy_server.dat"
            create => "true",
            edit_line => append_to_repo_file("rudder");

    commands:
        "/opt/rudder/bin/cf-agent -KI -D force_inventory";
        "/etc/init.d/cf-serverd stop";
        "/etc/init.d/cf-execd stop";
}

bundle edit_line append_to_repo_file(list) {
    insert_lines:
        "$(list)";
}

body delete tidyfiles_ {
     rmdirs     => "true";
     dirlinks   => "delete";
}

body classes repaired_(new_class) {
    promise_repaired    => { "$(new_class)"};
}

body package_method yum_rpm_sta {
  package_changes => "bulk";
  package_list_command => "/bin/rpm -qa --qf '%{name} %{version}-%{release} %{arch}\n'";
  package_patch_list_command => "/usr/bin/yum --quiet check-update";

  package_list_name_regex    => "^(\S+?)\s\S+?\s\S+$";
  package_list_version_regex => "^\S+?\s(\S+?)\s\S+$";
  package_list_arch_regex    => "^\S+?\s\S+?\s(\S+)$";

  package_installed_regex => ".*";
  package_name_convention => "$(name)";

  # set it to "0" to avoid caching of list during upgrade
  package_list_update_command => "/usr/bin/yum --quiet check-update";
  package_list_update_ifelapsed => "240";

  package_patch_installed_regex => "^\s.*";
  package_patch_name_regex    => "([^.]+).*";
  package_patch_version_regex => "[^\s]\s+([^\s]+).*";
  package_patch_arch_regex    => "[^.]+\.([^\s]+).*";

  package_add_command    => "/usr/bin/yum -y install";
  package_update_command => "/usr/bin/yum -y update";
  package_patch_command => "/usr/bin/yum -y update";
  package_delete_command => "/bin/rpm -e --nodeps --allmatches";
  package_verify_command => "/bin/rpm -V";
}

Let's see the code in details.

First install the repo file and install rudder-agent package:

    vars:
        container::
            "repo_content" string => "[Rudder]
name=Rudder 2.7 Repository
baseurl=http://repos/rudder/test/RHEL_6/
gpgcheck=0
";

    files:
        container::
            "/etc/yum.repos.d/rudder.repo"
                create => "true",
                edit_line => append_to_repo_file("$(repo_content)");

    packages:
        container::
            "rudder-agent"
                package_policy => "add",
                package_method => yum_rpm_sta,
                classes => repaired_("kick_inventory");

Then if package has been installed, call the bundle rudder_config:

methods:
        kick_inventory::
            "any" usebundle => rudder_config;

Populate /var/rudder/cfengine-community/policy_server.dat with your Rudder server name rudder then run the Rudder agent with force_inventory class to force node to send its inventory to the server, then stop running cfengine processes.

bundle agent rudder_config {
    files:
        "/var/rudder/cfengine-community/policy_server.dat"
            create => "true",
            edit_line => append_to_repo_file("rudder");

    commands:
        "/opt/rudder/bin/cf-agent -KI -D force_inventory";
        "/etc/init.d/cf-serverd stop";
        "/etc/init.d/cf-execd stop";
}

Everything else is cfengine functions to make all that work.
Finally use cf-agent -b rudder_install -D container -f /path/to/the/promise.cf to actually run this promise.

Rudder agent sets a cron job in /etc/cron.d/rudder-agent which should take care of starting itself in the following 5 minutes.

Grsecurity 3.10.10 kernel available

2013-09-01T00:00:00+02:00

Grsecurity test kernel is now available as RPM in the repository
You can fetch the SRPM here

Changelog:

- Update to kernel 3.10.10
- Update to grsecurity-2.9.1-3.10.10-201309011630.patch

At the same time, zfs modules have been recompiled for it.
You can also install them by using the yum repository.

Speed up Rudder new node's detection

2013-08-27T00:00:00+02:00

By default, a new node sending its inventory to the Rudder server, won't appear in the "Accept new nodes" menu before the next cf-agent run on the server.
In the worst case scenario, you'll have to wait 5 minutes for the node to appear. My issue was that i needed this process to be as quick as possible as i'm deploying rudder-agent through a kickstart post-install script, and i whished for the deployed node to have all the rudder rules applied before reboot.

As the inventory creation can be tracked in /var/rudder/inventories/incoming, a small python script using pyinotify to the rescue.

#!/usr/bin/python
import os, datetime, subprocess, re, thread
import pyinotify

command = "/opt/rudder/bin/cf-agent -K -b sendInventoryToCmdb -f /var/rudder/cfengine-community/inputs/promises.cf"
report_re = re.compile('R:.*')

wm = pyinotify.WatchManager() # Watch Manager
mask = pyinotify.IN_DELETE | pyinotify.IN_CREATE # watched events

def exec_proc(file, dummy1):
    process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=None, shell=True)
    output = process.communicate()
    date = datetime.datetime.now()
    report_line = report_re.search(output[0])
    print "%s: %s - %s" % (date, file, report_line.group())

class EventHandler(pyinotify.ProcessEvent):
    def process_IN_CREATE(self, event):
        date = datetime.datetime.now()
        print "%s: %s - %s created" % (date, event.path, event.name)

        if "incoming" in event.path:
            dummy_tup = event.name, 'null'
            thread.start_new_thread(exec_proc, dummy_tup)

    def process_IN_DELETE(self, event):
        date = datetime.datetime.now()
        print "%s: %s - %s deleted" % (date, event.path, event.name)


handler = EventHandler()
notifier = pyinotify.Notifier(wm, handler)
wdd = wm.add_watch('/var/rudder/inventories/', mask, rec=True)

notifier.loop()

So what performance gain do we have ?

# python bin/rudder_inv_notify.py 
2013-08-27 17:26:51.051183: /var/rudder/inventories/incoming - itclient-01-2013-09-04-17-26-46.ocs created
2013-08-27 17:26:54.616183: p4itclient-01-2013-09-04-17-26-46.ocs - R: @@DistributePolicy@@result_success@@root-DP@@root-distributePolicy@@20@@Send inventories to CMDB@@None@@2013-09-04 17:26:54+02:00##root@#Incoming inventories were successfully added to Rudder
2013-08-27 17:26:54.932790: /var/rudder/inventories/historical - 9530ce39-6699-49bc-819e-d8b72ae35a55 created

Around 3 seconds to have the node available in pending list. This let us use API to accept it ... I'll detail this further in another post.

Grsecurity 3.10.9 kernel available

2013-08-25T00:00:00+02:00

Grsecurity test kernel is now available as RPM in the repository
You can fetch the SRPM here

Changelog:

- Update to kernel 3.10.9
- Update to grsecurity-2.9.1-3.10.9-201308202015.patch
- Add variables for easier version update
- Add config CONFIG_SECCOMP

At the same time, zfs modules have been updated to 0.6.2.
You can also install them by using the yum repository.