#Building ARM cluster Part 2: Create and write system image with goback!

0. List of contents

1. Preface

Most of Raspberry like ARM devices have system image written to sdcard. Whenever we need to update base image we need to take out memory stick, write image and put it back. Now imagine that you have 8 boards and you want to update all of them (e.g your current system is crashing). So, you won’t do it manually, will you?

In this article I’m going to present whole process of creating base image and installing it via network. I have automated process of imaging by witting goback. It works in similar way to expect language.

You might ask “Why don’t you use goexpect?”. Answer is simple, it didn’t work for me in this particular case.


2. Create Ubuntu/Fedora system image

At first we start with creating empty raw image:

dd if=/dev/zero of=2g.img bs=1024k seek=2048 count=0

Then format:

fdisk 2g.img (o,p,enter,enter,enter, w)

Next create partitions and mount it:

sudo kpartx -a 2g.img
sudo mkfs.ext4 /dev/mapper/loop0p1
sudo mount /dev/mapper/loop0p1 rootfs/

Note: in above snippet I’m creating only one partition. In many cases you’ll have to create extra partition for /boot – it’s totally dependent from your uboot configuration. Usually, on such partition uboot, (z/u)Image and initrd is placed (those files are required for boot process). If you need two partitions you may do as follows:

fdisk 2g.img
input: o then p, select 1 partition, press enter to accept default first sector and pass +32M for the last sector
input: t and e
imput: n,p, select 2 partition, enter, enter
input: w

# Formatting
sudo mkfs.vfat /dev/mapper/loop0p1
sudo mkfs.ext4 /dev/mapper/loop0p2

This will create 32MB FAT16 boot partition where you should place your kernel with initrd. Of course you may create ext4 partition (which is done in fedora images by default) instead fat16 but check your uboot settings first (or change them):

fatload - for fat
ext4load - for ext4

Next step is to extract rootfs to raw image. Where can I find rootfs tarball?
Usually, board vendors are releasing system images with built-in kernel. You may simply copy rootfs from “ready” image to your own:

wget http://ftp.astral.ro/mirrors/fedora/pub/fedora/linux/releases/21/Images/armhfp/Fedora-Minimal-armhfp-21-5-sda.raw.xz

# Extract raw image
xzcat Fedora-Minimal-armhfp-21-5-sda.raw.xz | pv | dd of=fedora21.img bs=2M

# Mount
sudo kpartx -a fedora21.img
sudo mount /dev/mapper/loop0p2 rootfs/
rm -Rf /boot/* # you may also remove /lib/modules/$kern-ver and firmare if you wish

# Create rootfs
cd rootfs/ && tar -czvf ../fedora21.rootfs.tar.gz .

You may also create your own minimal image with debbootstrap or pacstrap (Archlinux).
When you are finished with creating your own rootfs umount and compress image.

mount rootfs
sudo kpartx -d 2g.img
dd if=2g.img bs=16M | pv | xz -9 - > 2g.img.xz

Wow, you have your own base image! (Without kernel)


3. Build kernel

Now it’s time to build your own kernel. This operation is specific to boards. The kernel for odroid, wandboard, parallella will be very different but the process is almost the same.

This example shows how to create image for odroid u3. I’m using this process for all of boards, replacing the kernel source (or as you will see in next section with buildroot)

# Download toolchain
sudo mkdir ~/toolchains
wget releases.linaro.org/13.04/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.7-2013.04-20130415_linux.tar.bz2
sudo tar jxvf gcc-linaro-arm-linux-gnueabihf-4.7-2012.12-20121214_linux.tar.bz2 -C ~/toolchains/

# Set variables
export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-
export PATH=~/toolchains/gcc-linaro-arm-linux-gnueabihf-4.7-2013.04-20130415_linux/bin:$PATH

# Build kernel
git clone --depth 1 github.com/hardkernel/linux.git -b odroid-3.8.y
cd linux
make odroidu_defconfig # change for e.g wandboard_defconfig
export ver=`make kernelversion`

# Copy compiled kernel with modules
mkdir output
cp .config output/config-$ver
cp arch/arm/boot/zImage output/
sudo make modules_install ARCH=arm INSTALL_MOD_PATH=./output
cd output
tar czvf modules.tar.gz lib/*
sudo rm -R lib
scp * root@odroid1:~/

Now create initrd on one of currently running machines:

ssh root@odroid0
mount /dev/mmcblk0p1 /mnt
cp zImage /mnt/
cp config-$ver /boot
tar xvf modules.tar.gz -C /
update-initramfs -c -k $ver
mkimage -A arm -O linux -T ramdisk -C none -a 0 -e 0 -n uInitrd -d /boot/initrd.img-$ver /boot/uInitrd-$ver

# Update
cp /boot/uInitrd-$ver /mnt/uInitrd

Generated uInitrd, kernel config, kernel and modules you may copy to your fresh image (e.g 2g.img)

Vioula you have created your first fully functional base image!


4. Buildroot

You’ve seen how much work you have to put in building your base image. But what if that process could be automated?
Article #Embedded development with Qemu, Beagleboard, Yocto, Ångström, Buildroot. Where to begin? contains some useful IMO information for those who has never used it before.

My forked version of buildroot contains some extra modifications for wandboard, odroid and parallella. If you have one of these boards you may use this repo, otherwise feel free to use mainline.

mkaczanowski – github buildroot

How to compile kernel using buildroot?

git clone https://github.com/mkaczanowski/buildroot
cd buildroot

# Choose board specific config
make odroidu_config
make linux

# Copy kernel
cp output/images/zImage ....

To make your own modifications use:

make xconfig
make linux-xconfig

This method is obviously more convenient and faster than doing it manually.


5. Uboot

Uboot is primirary booltloader used with ARM boards. MLO and X-loader are oftenly used in omap boards. Either way you’ll need it.

Uboot installation depends on memory type that you have, if you use

  • Sdcard
    sudo dd if=u-boot.img of=/dev/<MICROSD_DEVICE_NAME> bs=512 seek=2
  • Emmc flash memory has boot partitions, so you’ll have to use script such as sd_fusing.sh


Buildroot is also able to compile uboot for you. Try it!


6. Network imaging

Network imaging works as the following:

  1. Enter boot console
  2. Load lightweight system image to RAM
  3. Download base image and write it to sdcard/emmc
  4. Run postinstall operations (optional)

“lightweight system image” = kernel + ramdisk containing very little of tools (such as curl, resize2fs, e2fsprogs etc).

In my github buildroot fork, you may find:

  1. odroidu_minimal_defconfig
  2. parallella_defconfig
  3. wandboard_defconfig

which contains thin kernel configuration for specific boards. As the output you will get zImage with merged ramdisk. Copy it to your tftp server.


7. Odroid modifications

Above steps worked fine for wandboard and paralla,but odroid requires a bit more modifications:

  1. Uboot – odroid uses smsc95xx ethernet. As it came out, it uses common usb bus for communication. As hardkernel is using one uboot for each board type, they turned off ethernet for odroid u3.

    Odroid U3 uboot-usbnet-enabled

    Above branch contains patched and working usbnet enabled uboot version

  2. Kernel patch for smsc95xx – with this patch you’re able to pass “macaddr=<your_hw_addr>” as kernel arg and you will finally have static mac address (without changing /etc/smsc*). Author of patch: Danny Kukawka

    MAC addr kernel patch


8. Parallella modifications

By default parallella has disabled UART. So we’ll have to enable it by modyfing device tree:

dtc -I dtb -O dts -o devicetree.dts devicetree.dtb
dtc -I dtb -O dts -o devicetree.dts devicetree.dtb

This file is a dts file which enables serial output.


9. Uboot boot parameters

Odroid && Wandboard:

setenv ethact sms0
setenv ethaddr 00:10:75:2A:AE:E0
setenv gatewayip
setenv netmask
setenv serverip
setenv usbethaddr 00:10:75:2A:AE:E0
setenv ipaddr
setenv bootargs console=${console},${baudrate} ${optargs} root=/dev/ram video=${video}
usb start
tftp 0x40008000 odroid/zImage


setenv ethaddr 00:10:75:2A:AE:E0
setenv gatewayip
setenv netmask
setenv serverip
setenv usbethaddr 00:10:75:2A:AE:E0
setenv ipaddr
tftp 0x4000000 parallella/parallella.bit.bin
fpga load 0 0x4000000 0x3dbafc
tftp 0x3000000 parallella/uImage
tftp 0x2A00000 parallella/devicetree.dtb
tftp 0x1100000 parallella/initrd
bootm 0x3000000 0x1100000 0x2A00000

Executing above commands will load system in memory and run it.

What these 0x4000000, 0x40008000, 0x2A00000 etc. adresses means?

Complete explanation you may find in: here

“On 32-bit systems, memory is now divided into ”high” and ”low” memory. Low memory continues to be mapped directly into the kernel’s address space, and is thus always reachable via a kernel-space pointer. High memory, instead, has no direct kernel mapping. When the kernel needs to work with a page in high memory, it must explicitly set up a special page table to map it into the kernel‘s address space first. This operation can be expensive, and there are limits on the number of high-memory pages which can be mapped at any particular time.”

So let’s look at our configuration. Odroid U3 has 2GB of RAM memory. Where 0x40008000 address is pointing to?
Max Ram capacity (2GB) = 2147483648 = 0x80000000 (hex)
Kernel LoadAddr = 0x40008000 = 1073774592 / 1024 / 1024 = 1024.03125 (MB) = ~1GB

As you see, this address is pointing to approximately the middle of the memory.
On the other hand parallella has only 1GB of memory an additional fpga binaries to load, so the address distribution looks fairly different.

On 4GB RAM device, memory will be split in 3G/1G manner.
Address distribution presented above is not a rule, but you may encounter this ratio very often.


10. Imaging with goback

Goback github repository

To put it all together I decided to create project named goback. The main idea is to automate imaging.
Program uses serial console to interact with device. So it is extreemly important to have proper ttySAC0, ttyPS0, ttyS0 enabled.

How does it work?:
1. Loads “steps” from config (StepList struct is a linked-list)
2. Read lines from serial line
3. If line contains “Step.Expect” or “Step.Trigger” phrase, then program will continue to the next step (Expect) or will execute onTrigger (Trigger)
4. Program finishes when it reaches the end of the list or there is an error

Let’s look into example steps:

        stEnterUboot := &step.Step{
                Trigger: "ModeKey Check...",
                OnTrigger: func() {
                        util.MustSendCmd(w, "\n", true)
                Expect:  "Exynos4412 #",
                Msg:     "Enter u-boot console",
                Timeout: 10 * time.Second,

        stStartEthernet := &step.Step{
                OnTrigger: func() {
                        // Double \n -> uboot bug
                        util.MustSendCmd(w, "usb start", true)
                Expect:    "1 Ethernet Device(s) found",
                Msg:       "USB Ethernet start",
                SendProbe: true,

stEnterUboot enters uboot console, when “ModeKey Check…” text appears then it is a right time to send “enter” to stop countdown. When “Exynos4412 #” text appears it means that we are logged in console.

stStartEthernet starts usbnet ethernet in uboot. When “1 Ethernet Device(s) found” appears it means that eth is ready to use.

Note: Now it is a hacky project, but if you wish to add your configuration feel free to modify config and flasher.go 🙂

How to use it?

Usage of ./goback:
  -action="flash": What do you want to do? [flash, power_[on|off|switch]]
  -boards="": List boards to flash
  -debug=false: Set true to print serial console output
  -system="ubuntu": Choose system [Ubuntu|Fedora]


Important: tty.conf contains json map of ttyUSB for each machine. Don’t forget to modify it.

11. Demo

12. Prebuilt images

TFTP leightweight system images for parallella, odroid u3, wandboard quad

Ubuntu U3/U2 docker enabled + fedora 21

Leave a Comment.