#Building ARM cluster Part 3: Docker, fleet, etcd. Distribute containers!

0. List of contents

1. Preface

This 3rd article is going to cover post-installation process, where we’ll install golang + docker +fleet + etcd. As most of these tools is made for 64bit systems we will have to modify them a bit. The set of tools is used in coreos for managing containers in a cluster, so that is a perfect tool for us!


2. Post-install: ResizeFS & move configs

In previous article: #Building ARM cluster Part 2: Create and write system image with goback! we have managed to write system image to sd/emmc memory via network. Now it is time to make system to serve some services.

Installed image is 2GB size, probably your memory stick is much larger than that, so you’ll need to resize system partition. You may do it by this simple script:

#! / Bin / sh
parted /dev/mmcblk0 <<EOF
unit chs
rm 2
mkpart primary 13,184,18 1005,63,31

After reboot execute and run e2fsck:

resize2fs /dev/mmcblk0p2

Where 13,184,18 is where partition starts, 1005,63,31 partition end. Check your configuration by:

parted /dev/mmcblk0 unit chs print

Installing configuration (/etc) files

After re-image each file you have modified disappears. In order to recover them after image is written I’m using the simplest possible approach = store /etc/* in repository. Here are some files from my private repository:



ExecStart = / usr / bin / fleetd




Wants=network.target network-online.target
After=network.target network-online.target cloudinit.service

ExecStart = / usr / bin / DCE





Type = oneshot
ExecStart=/usr/bin/coreos-cloudinit --from-file /root/cluster/cloud-init/odroid.conf





    name: $host
    data_dir: /var/lib/etcd/
    discovery: https://discovery.etcd.io/da3221e6c34261892a8c89f1845631b6
    addr: $private_ipv4:4001
    peer-addr: $private_ipv4:7001
    public-ip: $private_ipv4   # used for fleetctl ssh command



verbosity = 2


Custom postinstall commands:

echo "[Step] Drop iptables"
systemctl disable firewalld.service
systemctl stop firewalld.service

echo "[Step] Run yum"
yum check-update
yum install -y python-pip vim parted-devel mercurial git-core net-tools tar gcc wget nfs-utils
pip install reparted


3. Install go, docker, fleet, etcd, cloudinit

At first install go.

echo "[Step] Install go"
cd /usr/src/
# Install go
git clone https://go.googlesource.com/go
cd go
git checkout go1.4.1
cd src
echo "export PATH=$PATH:/usr/src/go/bin/" >> ~/.bashrc
echo "export GOPATH = / usr / src / spouse" >> ~ / .bashrc
export PATH=$PATH:/usr/src/go/bin/
export GOPATH = / usr / src / spouse
mkdir / usr / src / spouse

Then install docker (on fedora it is a bit tricky). If you’re using ubuntu then download docker.io from repo. Otherwise:

echo "[Step] Install Docker"
# Install docker
mkdir / usr / src / docker-install
cd / usr / src / docker-install
yum install -y rpm-build glibc-static
rpmbuild --rebuild http://copr-be.cloud.fedoraproject.org/results/gipawu/kernel-aufs/fedora-21-x86_64/aufs-util-3.9-1.fc20/aufs-util-3.9-1.fc21.src.rpm
rpm -i /usr/src/docker-install/rpmbuild/RPMS/armv7hl/aufs-util-3.9-1.fc21.armv7hl.rpm
yum install -y bridge-utils device-mapper device-mapper-libs libsqlite3x docker-registry docker-storage-setup
mkdir / usr / src / docker
cd / usr / src / docker
wget https://kojipkgs.fedoraproject.org//packages/docker-io/1.5.0/18.git92e632c.fc23/src/docker-io-1.5.0-18.git92e632c.fc23.src.rpm
rpm2cpio docker-io-1.5.0-18.git92e632c.fc23.src.rpm | cpio -idmv
tar -xzf docker-92e632c.tar.gz
curl -L https://github.com/umiddelb/armhf/raw/master/bin/docker-1.5.0> docker

install -p -m 755 docker / usr / bin / docker

# install bash completion
install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/completion/bash/docker /usr/share/bash-completion/completions

# install container logrotate cron script
install -p -m 755 docker-logrotate.sh /etc/cron.daily/docker-logrotate

# install vim syntax highlighting
install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/syntax/vim/doc/dockerfile.txt /usr/share/vim/vimfiles/doc
install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/syntax/vim/ftdetect/dockerfile.vim /usr/share/vim/vimfiles/ftdetect
install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/syntax/vim/syntax/dockerfile.vim /usr/share/vim/vimfiles/syntax

# install udev rules
install -p docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/udev/80-docker.rules /etc/udev/rules.d

# Install systemd / init scripts
install -p -m 644 docker.service /usr/lib/systemd/system

# for additional args
install -p -m 644 docker.sysconfig / etc / sysconfig / docker
install -p -m 644-docker network.sysconfig / etc / sysconfig / network-docker
install -p -m 644-docker storage.sysconfig / etc / sysconfig / docker-storage

# install docker config directory
sudo install -dp / etc / docker

getent passwd dockerroot > /dev/null || sudo /usr/sbin/useradd -r -d /var/lib/docker -s /sbin/nologin -c "Docker User" dockerroot
systemctl enable docker.service
systemctl enable docker.socket


Install fleet:

echo "[Step] Install fleet"
cd /usr/src/
go get golang.org/x/tools/cmd/cover
git clone https://github.com/coreos/fleet.git
cd fleet
ln -s /usr/src/fleet/bin/* /usr/bin/
source ~ / .bashrc


Install etcd with patches:

# Install etcd
echo "[Step] Install etcd"
curl -sSL -k https://github.com/coreos/etcd/archive/v$ETCDI_VERSION.tar.gz | tar --touch -v -C /usr/src -xz
cd /usr/src/etcd-$ETCDI_VERSION
curl https://raw.githubusercontent.com/mkaczanowski/docker-archlinux-arm/master/archlinux-etcd/patches/raft.go.patch > raft.go.patch
curl https://raw.githubusercontent.com/mkaczanowski/docker-archlinux-arm/master/archlinux-etcd/patches/server.go.patch > server.go.patch
curl https://raw.githubusercontent.com/mkaczanowski/docker-archlinux-arm/master/archlinux-etcd/patches/watcher_hub.go.patch > watcher_hub.go.patch
patch etcdserver/raft.go < raft.go.patch
patch etcdserver/server.go < server.go.patch
patch store/watcher_hub.go < watcher_hub.go.patch
ln -s /usr/src/etcd-$ETCDI_VERSION/bin/* /usr/bin/
mkdir / var / lib / etcd

systemctl enable cloudinit
systemctl enable etcd
systemctl enable fleetd


Install cloudinit and replace variables $2 = hostname, $3 = host ipaddr

echo "[Step] Cloud config"
cd /usr/src/
git clone https://github.com/coreos/coreos-cloudinit.git
cd coreos-cloudinit
ln -s /usr/src/coreos-cloudinit/bin/coreos-cloudinit /usr/bin/coreos-cloudinit
sed -i "s/\$private_ipv4/$3/g" /usr/src/cluster/cloud-init/odroid.conf
sed -i "s/\$private_ipv4/$3/g" /etc/fleet/fleet.conf
sed -i "s/\$host/$2/g" /usr/src/cluster/cloud-init/odroid.conf


4. Build archlinux images

Building docker images is pretty simple. As ArchlinuxARM is very lightweight system I’m using it as base image for further services in docker. Systemd is enabled in all images.

Archlinux ARM DockerHub

Docker image source


How to build nginx container?

FROM mkaczanowski/archlinux-systemd
MAINTAINER Mateusz Kaczanowski, kaczanowski.mateusz@gmail.com

RUN pacman -Syu --noconfirm nginx; systemctl enable nginx
RUN pacman -Scc --noconfirm; 
RUN mkdir -p /etc/nginx/conf.d
RUN mkdir -p /etc/nginx/sites-enabled
ADD ./config/nginx.service /etc/systemd/system/multi-user.target.wants/nginx.service
ADD ./config/nginx.conf /etc/nginx/nginx.conf
ADD ./config/00-default.conf /etc/nginx/sites-enabled/00-default.conf

ONBUILD RUN rm /etc/nginx/sites-enabled/00-default.conf
ONBUILD ADD ./sites/*.conf /etc/nginx/sites-enabled/

VOLUME ["/srv/http/"]


CMD ["/usr/lib/systemd/systemd"]


Run container:

CONTAINER=$(sudo docker run --privileged -d -v /srv/http:/srv/http -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 80:80 -p 443:443 mkaczanowski/archlinux-nginx)


5. Run nginx container with fleet

Eventually we are able to run nginx container on all of machines at once using fleet!

Nginx service:


ExecStartPre=-/bin/docker kill nginx
ExecStartPre=-/bin/docker rm nginx
ExecStartPre=/bin/docker pull mkaczanowski/archlinux-nginx
ExecStart=/bin/docker run --name nginx --privileged -v /srv/http:/srv/http -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 80:80 -p 443:443 mkaczanowski/archlinux-nginx
ExecStop=/bin/docker stop nginx


Now using fleet we may execute container:

fleetctl --endpoint= submit nginx
fleetctl --endpoint= start nginx

It works! Now you have fully managed cluster!




  1. I suppose that you don’t install fleet/etcd on each machine and use docker images you’ve built instead, and I also noticed that there is no image for cloud-config. How do you manage cluster configuration when running etcd/fleet in containers?

Leave a Comment.