Intro

This 3rd article is going to cover post-installation process, where we’ll install golang + docker +fleet + etcd. As most of these tools is made for 64bit systems we will have to modify them a bit. The set of tools is used in coreos for managing containers in a cluster, so that is a perfect tool for us!

Post-install: ResizeFS & move configs

In previous article: #Building ARM cluster Part 2: Create and write system image with goback! we have managed to write system image to sd/emmc memory via network. Now it is time to make system to serve some services.

Installed image is 2GB size, probably your memory stick is much larger than that, so you’ll need to resize system partition. You may do it by this simple script:

#!/bin/sh
parted /dev/mmcblk0 <

After reboot execute and run e2fsck:resize2fs /dev/mmcblk0p2

Where 13,184,18 is where partition starts, 1005,63,31 partition end. Check your configuration by:parted /dev/mmcblk0 unit chs print

Installing configuration (/etc) files

After re-image each file you have modified disappears. In order to recover them after image is written I’m using the simplest possible approach = store /etc/* in repository.
Here are some files from my private repository:

fleetd.service

[Unit]
Description=fleetd
After=etcd.service

[Service]
ExecStart = / usr / bin / fleetd
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

etcd.service

[Unit]
Description=etcd
Wants=network.target network-online.target
After=network.target network-online.target cloudinit.service

[Service]
ExecStart = / usr / bin / DCE
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

cloudinit.service

[Unit]
Description=cloudinit

[Service]
Type = oneshot
ExecStart=/usr/bin/coreos-cloudinit --from-file /root/cluster/cloud-init/odroid.conf

[Install]
WantedBy=multi-user.target

odroid.conf

#cloud-config

Core:
  etcd:
    name: $host
    data_dir: /var/lib/etcd/
    discovery: https://discovery.etcd.io/da3221e6c34261892a8c89f1845631b6
    addr: $private_ipv4:4001
    peer-addr: $private_ipv4:7001
  fleet:
    public-ip: $private_ipv4   # used for fleetctl ssh command

fleet.conf

verbosity = 2
etcd_servers=["http://$private_ipv4:4001"]
etcd_request_timeout=30.0
public_ip="$private_ipv4"

Custom postinstall commands:

echo "[Step] Drop iptables"
systemctl disable firewalld.service
systemctl stop firewalld.service

echo "[Step] Run yum"
yum check-update
yum install -y python-pip vim parted-devel mercurial git-core net-tools tar gcc wget nfs-utils
pip install reparted

Install go, docker, fleet, etcd, cloudinit

At first install go.

echo "[Step] Install go"
cd /usr/src/
# Install go
git clone https://go.googlesource.com/go
cd go
git checkout go1.4.1
cd src
./make.bash
echo "export PATH=$PATH:/usr/src/go/bin/" >> ~/.bashrc
echo "export GOPATH = / usr / src / spouse" >> ~ / .bashrc
export PATH=$PATH:/usr/src/go/bin/
export GOPATH = / usr / src / spouse
mkdir / usr / src / spouse

Then install docker (on fedora it is a bit tricky). If you’re using ubuntu then download docker.io from repo. Otherwise:

echo "[Step] Install Docker"
# Install docker
mkdir / usr / src / docker-install
cd / usr / src / docker-install
yum install -y rpm-build glibc-static

rpmbuild --rebuild http://copr-be.cloud.fedoraproject.org/results/gipawu/kernel-aufs/fedora-21-x86_64/aufs-util-3.9-1.fc20/aufs-util-3.9-1.fc21.src.rpm
rpm -i /usr/src/docker-install/rpmbuild/RPMS/armv7hl/aufs-util-3.9-1.fc21.armv7hl.rpm

yum install -y bridge-utils device-mapper device-mapper-libs libsqlite3x docker-registry docker-storage-setup

mkdir / usr / src / docker
cd / usr / src / docker
wget https://kojipkgs.fedoraproject.org//packages/docker-io/1.5.0/18.git92e632c.fc23/src/docker-io-1.5.0-18.git92e632c.fc23.src.rpm

rpm2cpio docker-io-1.5.0-18.git92e632c.fc23.src.rpm | cpio -idmv
tar -xzf docker-92e632c.tar.gz
curl -L https://github.com/umiddelb/armhf/raw/master/bin/docker-1.5.0> docker

### INSTALL
install -p -m 755 docker / usr / bin / docker

# install bash completion
install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/completion/bash/docker /usr/share/bash-completion/completions

# install container logrotate cron script
install -p -m 755 docker-logrotate.sh /etc/cron.daily/docker-logrotate

# install vim syntax highlighting
install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/syntax/vim/doc/dockerfile.txt /usr/share/vim/vimfiles/doc
install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/syntax/vim/ftdetect/dockerfile.vim /usr/share/vim/vimfiles/ftdetect
install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/syntax/vim/syntax/dockerfile.vim /usr/share/vim/vimfiles/syntax

# install udev rules
install -p docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/udev/80-docker.rules /etc/udev/rules.d

# Install systemd / init scripts
install -p -m 644 docker.service /usr/lib/systemd/system

# for additional args
install -p -m 644 docker.sysconfig / etc / sysconfig / docker
install -p -m 644-docker network.sysconfig / etc / sysconfig / network-docker
install -p -m 644-docker storage.sysconfig / etc / sysconfig / docker-storage

# install docker config directory
sudo install -dp / etc / docker

getent passwd dockerroot > /dev/null || sudo /usr/sbin/useradd -r -d /var/lib/docker -s /sbin/nologin -c "Docker User" dockerroot
systemctl enable docker.service
systemctl enable docker.socket

Install fleet:

echo "[Step] Install fleet"
cd /usr/src/
go get golang.org/x/tools/cmd/cover
git clone https://github.com/coreos/fleet.git
cd fleet
./build
./test
ln -s /usr/src/fleet/bin/* /usr/bin/
source ~ / .bashrc

Install etcd with patches:

# Install etcd
echo "[Step] Install etcd"
ETCDI_VERSION=2.0.4
curl -sSL -k https://github.com/coreos/etcd/archive/v$ETCDI_VERSION.tar.gz | tar --touch -v -C /usr/src -xz
cd /usr/src/etcd-$ETCDI_VERSION

curl https://raw.githubusercontent.com/mkaczanowski/docker-archlinux-arm/master/archlinux-etcd/patches/raft.go.patch > raft.go.patch
curl https://raw.githubusercontent.com/mkaczanowski/docker-archlinux-arm/master/archlinux-etcd/patches/server.go.patch > server.go.patch
curl https://raw.githubusercontent.com/mkaczanowski/docker-archlinux-arm/master/archlinux-etcd/patches/watcher_hub.go.patch > watcher_hub.go.patch

patch etcdserver/raft.go < raft.go.patch
patch etcdserver/server.go < server.go.patch
patch store/watcher_hub.go < watcher_hub.go.patch
./build
ln -s /usr/src/etcd-$ETCDI_VERSION/bin/* /usr/bin/
mkdir / var / lib / etcd

systemctl enable cloudinit
systemctl enable etcd
systemctl enable fleetd

Install cloudinit and replace variables $2 = hostname, $3 = host ipaddr

echo "[Step] Cloud config"
cd /usr/src/
git clone https://github.com/coreos/coreos-cloudinit.git
cd coreos-cloudinit
./build
ln -s /usr/src/coreos-cloudinit/bin/coreos-cloudinit /usr/bin/coreos-cloudinit
sed -i "s/\$private_ipv4/$3/g" /usr/src/cluster/cloud-init/odroid.conf
sed -i "s/\$private_ipv4/$3/g" /etc/fleet/fleet.conf
sed -i "s/\$host/$2/g" /usr/src/cluster/cloud-init/odroid.conf

Build archlinux images

Building docker images is pretty simple. As ArchlinuxARM is very lightweight system I'm using it as base image for further services in docker. Systemd is enabled in all images.

How to build nginx container?

FROM mkaczanowski/archlinux-systemd
MAINTAINER Mateusz Kaczanowski, [email protected]

RUN pacman -Syu --noconfirm nginx; systemctl enable nginx
RUN pacman -Scc --noconfirm; 
RUN mkdir -p /etc/nginx/conf.d
RUN mkdir -p /etc/nginx/sites-enabled
ADD ./config/nginx.service /etc/systemd/system/multi-user.target.wants/nginx.service
ADD ./config/nginx.conf /etc/nginx/nginx.conf
ADD ./config/00-default.conf /etc/nginx/sites-enabled/00-default.conf

ONBUILD RUN rm /etc/nginx/sites-enabled/00-default.conf
ONBUILD ADD ./sites/*.conf /etc/nginx/sites-enabled/

VOLUME ["/srv/http/"]

EXPOSE 80
EXPOSE 443

CMD ["/usr/lib/systemd/systemd"]

Run container:

CONTAINER=$(sudo docker run --privileged -d -v /srv/http:/srv/http -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 80:80 -p 443:443 mkaczanowski/archlinux-nginx)

Run nginx container with fleet

Eventually we are able to run nginx container on all of machines at once using fleet!

Nginx service:

[Unit]
Description=nginx
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
ExecStartPre=-/bin/docker kill nginx
ExecStartPre=-/bin/docker rm nginx
ExecStartPre=/bin/docker pull mkaczanowski/archlinux-nginx
ExecStart=/bin/docker run --name nginx --privileged -v /srv/http:/srv/http -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 80:80 -p 443:443 mkaczanowski/archlinux-nginx
ExecStop=/bin/docker stop nginx

Now using fleet we may execute container:

fleetctl --endpoint=http://192.168.4.100:4001 submit nginx
fleetctl --endpoint=http://192.168.4.100:4001 start nginx

It works! Now you have fully managed cluster!

Creating software is what I love, it's what I care about, and what I strive to do better every day.

# Building ARM cluster Part 3: Docker, fleet, etcd. Distribute containers!