I'm using 5 x RockPro64 (80 USD) for bootstrapping a Kubernetes / K3s cluster, enabling CI builds for ARM apps and keeping a few lightweight services up.
Here's how I've installed NixOS on it:
1. NixOS ARM & U-Boot version
Ideally use a newer image version, like 20.09 or unstable. But I'll be using an older version 20.03: 2020-08-31 19:09:22 because updating version is a moving target - latest is (usually) better, the important part is to understand the process, and then you get to decide which NixOS and U-boot version you want.
2 Acquire NixOS for Aarch64 image
2.1. Option 1: Build from sources
Important: Requires an aarch64 host for building it!
$ git clone https://github.com/NixOS/nixpkgs.git; cd nixpkgs
$ nix-build nixos -I nixos-config=nixos/modules/installer/sd-card/sd-image-aarch64.nix -A config.system.build.sdImage
$ ls -l ./result/sd-image/nixos-sd-image-21.11pre-git-aarch64-linux.img.zst
2.1. Option 2: Download from Hydra
2.1.2.1 Pick NixOS version
$ wget https://hydra.nixos.org/build/126272499/download/1/nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2 # (653.27 MiB)
2.1.2.2 Check for corruption
$ sha256sum nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2
5613c806ddee0f62c7130780f31e40286296ae8b36a806c6f4f752a4f0c3a0892.2. Decompress file
For .xz: $ xz -d file_name.img.xz
For .zst: $ zstd -d file_name.img.zst
For .bz2: $ bzip2 -d nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2
Uncompressed "nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img" file (2,7G) is created.
2.3. Export my image location (to save typing):
$ export MYIMAGE=nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img3. Download U-Boot
Option 1: As a nix package using Nix (Nix should be already installed)
$ nix-env -i uboot-rockpro64-rk3399_defconfig
Option 2: From hydra: Pick U-Boot version
Troubleshooting:
Last time I tried U-boot latest was broken [error here]. If still broken, pin Uboot version as suggested bellow - it works.
From Hydra's nixpkg ubootRockPro64.aarch64-linux. Download build 126171011, in details tab, locate "Output store paths", valued as "/nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.07".
To download pinned version of U-Boot:
$ nix-env -i /nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.073.1. Export U-Boot location (to save typing):
$ export UBOOT_PATH=/nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.07/4. Delete unused first partition from NixOS ARM image
4.1. List partitions
$ parted ${MYIMAGE} print4.2. Delete partition 1
$ parted ${MYIMAGE} rm 14.2. Check if Partition 1 is gone.
$ parted ${MYIMAGE} print5. Copy U-Boot to image
$ dd if=${UBOOT_PATH}idbloader.img of=${MYIMAGE} conv=fsync,notrunc bs=512 seek=64
$ dd if=${UBOOT_PATH}u-boot.itb of=${MYIMAGE} conv=fsync,notrunc bs=512 seek=16384
6. Image overview [Optional]
6.1. Mount image.
$ fdisk -l ${MYIMAGE}Offset is 512 (Block size) * 77824 (Start), equals 39845888
Partition files are now accessible:
intj@nix-2700x:/mnt/myimage]$ tree -L 2
.
├── boot
│ ├── extlinux
│ └── nixos
├── lost+found [error opening dir]
├── nix
│ └── store
└── nix-path-registration
6 directories, 1 file7. Copy image to SDCard.
Find your SDCard device (or adapter). Be careful because on misidentification other device's data will be destroyed.
# lsblk -fIn my case it is "/dev/sde". Then, to write our image to it, replace 'of' with your SDCard device location:
# dd status=progress conv=fsync bs=1M if=${MYIMAGE} of=/dev/sde
# eject /dev/sde
8. Now test it
Plug your SDCard to your RockPro64! And turn it on!
8.1 Useful tips:
- RockPro64 is turned on when the white LED (close to USB ports) is turned on. If it is off, it has not powered on yet. [In that case I press reset button and usually works.]
- When powering off the RockPro64 board, remove power supply and wait a bit longer before powering it on again. Or it might not automatically power up when plugging back power and you'd have to press reset for powering it up.
- For 4k monitor/TV use HDMI 2.0 (or superior) cable.
9. NixOS Configuration
IMPORTANT: `configuration.nix` which is used to configure NixOS does not exist yet. The installation image does not include a configuration.nix
On booting the image, you can generate a configuration.nix with `nixos-generate-config`.
To configure your system edit `/etc/nixos/configuration.nix` and to activate the new configuration run `nixos-rebuild switch` (as root).
Use the NixOS manual as reference to configure your system.
Interesting bits:
`configuration.nix` is only needed for building the system. Booting only requires the files in /nix and /boot.
NixOS image itself is generated from https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/cd-dvd/sd-image-aarch64.nix.
10. Configure Network
10.1. At NixOS
To receive IP from local DHCP server.
# k#/configuration.nix
networking.hostName = "k#"; # Define your hostname.
networking.interfaces.eth0.useDHCP = true;
10.2. At Router
Sample configuration for Ubiquiti EdgeRouter but any BGP router (Mikrotik, etc) might be configured as well (with different syntax):
# Basic network configuration (define IPs, gateway, bridge ports, etc.):
interfaces {
ethernet eth0 {
address 192.168.4.2/30 # Uplink
}
switch switch0 {
address 192.168.51.254/24
switch-port {
interface eth1 {}
interface eth2 {}
interface eth3 {}
interface eth4 {}
interface eth5 {}
}
}
}
protocols {
static {
route 192.168.0.0/16 {
next-hop 192.168.4.1 {
description interno
}
}
}
}
# DHCP server configuration (to deliver IP address to hosts):
service {
dhcp-server {
disabled false
hostfile-update disable
shared-network-name dhcpd-bridge51 {
authoritative disable
subnet 192.168.51.0/24 {
default-router 192.168.51.254
dns-server 8.8.8.8
dns-server 8.8.4.4
lease 86400
start 192.168.51.10 {
stop 192.168.51.253
}
static-mapping h1 {
ip-address 192.168.51.1
mac-address 0e:19:50:50:70:6c
}
static-mapping h2 {
ip-address 192.168.51.2
mac-address 02:a6:8e:02:7d:c7
}
# Add a static-mapping register for each node.
}
}
}
}
11. K3s set-up
11.1. Configure master node (192.168.51.1)
{
boot.kernelModules = [ "overlay" "br_netfilter" ]; # Needed for K3s?
boot.kernel.sysctl = {
"net.bridge-nf-call-ip6tables" = 1;
"net.bridge-nf-call-iptables" = 1;
"net.ipv4.ip_forward" = 1;
};
services.k3s = {
enable = true;
role = "server";
extraFlags = "--no-deploy servicelb --no-deploy traefik --bind-address 192.168.51.1 --tls-san 192.168.51.1 --node-ip=192.168.51.1 --node-external-ip=192.168.51.1 --write-kubeconfig-mode 644";
};
}
11.2. Configure worker nodes (192.168.51.[2-5])
{
boot.kernelModules = [ "overlay" "br_netfilter" ]; # Needed for K3s?
boot.kernel.sysctl = {
"net.bridge-nf-call-ip6tables" = 1;
"net.bridge-nf-call-iptables" = 1;
"net.ipv4.ip_forward" = 1;
};
services.k3s = {
enable = true;
role = "agent";
serverAddr = "https://192.168.51.1:6443";
token = "FILL-WITH-K3S-MASTER-TOKEN"; # Token located at k3s-master's '/var/lib/rancher/k3s/server/node-token'
extraFlags = "--node-ip=192.168.51.2 --node-external-ip=192.168.51.2"; # Update worker IPs accordingly.
};
}
12. Accessing K3s cluster
Import kubeconfig file from Master:
$ scp k3s-master:/etc/rancher/k3s/k3s.yaml ~/.kube/
kubectl looks up configuration at "~/.kube/config" location. You can symlink k3s.yaml to config:
$ ln -s ~/.kube/k3s.yaml ~/.kube/config
To safeguard credentials set permission to no group/world access:
$ chmod 700 ~/.kube/k3s.yaml
Test access to cluster:
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k1 Ready control-plane,master 1h v1.20.5+k3s-355fff30-dirty 192.168.51.1 192.168.51.1 NixOS 21.05 (Okapi) 5.10.29 containerd://1.4.4-k3s1
k2 Ready 1h v1.20.5+k3s-355fff30-dirty 192.168.51.2 192.168.51.2 NixOS 21.05 (Okapi) 5.10.29 containerd://1.4.4-k3s1
k3 Ready 1h v1.20.5+k3s-355fff30-dirty 192.168.51.3 192.168.51.3 NixOS 21.05 (Okapi) 5.10.29 containerd://1.4.4-k3s1
k4 Ready 1h v1.20.5+k3s-355fff30-dirty 192.168.51.4 192.168.51.4 NixOS 21.05 (Okapi) 5.10.29 containerd://1.4.4-k3s1
k5 Ready 1h v1.20.5+k3s-355fff30-dirty 192.168.51.5 192.168.51.5 NixOS 21.05 (Okapi) 5.10.29 containerd://1.4.4-k3s1
13. MetalLB BGP
MetalLB layer 2 (ARP) load balancing works only for failover, it does not load balance traffic. Avoid it. Use BGP instead. Be aware of MetalLB BGP limitations like traffic is splitted evenly between nodes even when running multiple instances of pods in same node.
13.1. Configure router for BGP
Edge router sample configuration:
protocols {
bgp 64500 {
parameters {
router-id 192.168.51.254
}
maximum-paths {
ibgp 32
}
neighbor 192.168.51.1 {
remote-as 64501
soft-reconfiguration {
inbound
}
}
neighbor 192.168.51.2 {
remote-as 64501
soft-reconfiguration {
inbound
}
}
# Add a neighbor register for each node. I won't repeat it.
}
}
13.2. Install MetalLB
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/metallb.yaml
# On first install only
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
Sample MetalLB configuration for deliverying private IPs by default and public optionally using BGP:
# ./metallb-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
peers:
- peer-address: 192.168.51.254
peer-asn: 64500
my-asn: 64501
address-pools:
- name: default
protocol: bgp
auto-assign: true
avoid-buggy-ips: true
addresses:
- 192.168.128.0/19
- name: public
protocol: bgp
auto-assign: false
addresses:
- 189.113.69.96/29 # Change this to your public network!
- 179.191.211.24/29 # Change this to your public network!
Apply configuration:
$ kubectl apply -f ./metallb-config.yaml
At Router, neighbors will show:
ubnt@ubnt-k3s:~$ show ip bgp summary
BGP router identifier 192.168.51.254, local AS number 64500
BGP table version is 1
0 BGP AS-PATH entries
0 BGP community entries
1 Configured ebgp ECMP multipath: Currently set at 1
32 Configured ibgp ECMP multipath: Currently set at 32
Neighbor V AS MsgRcv MsgSen TblVer InQ OutQ Up/Down State/PfxRcd
192.168.51.1 4 64501 5 5 1 0 0 00:01:51 0
192.168.51.2 4 64501 5 5 1 0 0 00:01:51 0
192.168.51.3 4 64501 5 5 1 0 0 00:01:51 0
192.168.51.4 4 64501 5 5 1 0 0 00:01:51 0
192.168.51.5 4 64501 6 6 1 0 0 00:02:28 0
Total number of neighbors 5
Total number of Established sessions 5
Exposing a K8s Service will list as a connected route:
ubnt@ubnt-k3s:~$ show ip bgp ipv4 unicast
BGP table version is 8, local router ID is 192.168.51.254
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, l - labeled
S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 192.168.128.1/32 192.168.51.5 0 0 64501 ?
* 192.168.51.1 0 0 64501 ?
* 192.168.51.2 0 0 64501 ?
* 192.168.51.4 0 0 64501 ?
* 192.168.51.3 0 0 64501 ?
Total number of prefixes 1
13.3. Optional: Reset K3s host
# systemctl stop k3s
# rm -rf /etc/rancher/k3s/
# rm -rf /etc/rancher/node/
# rm -rf /var/lib/rancher/k3s/
# nixos-rebuild switch
14. Troubleshooting
14.1. Won't boot with eMMC.
Serial output as in picture. In brief:
mmc1: mmc_select_hs200 failed, error -110
mmc1: error -110 whilsting MMC created
mmc1: mmc_select_hs200 failed, error -110
error -110 whilstd
mmc1: mmc_select_hs200 failed, error -110
mmc1: error -110 whilst initialising MMC card
kbd_mode: KDSKBMODE: Inappropriate ioctl for device
...
An error occurred in stage 1 of the boot process, which must mount the root filesystem on `/mnt-root' and then start stage 2. Press one of the following keys:
r) to reboot immediately
*) to ignore the error and continue
If you press 'r', the system will boot correctly. But you don't want to press 'r' every boot. :)
@samueldr suspects it's a marginal behaviour in the Linux mmc driver around hs200, hs400, access mode.
Workaround
# Workaround kindly offered by @samueldr
Either:
1) unbind/bind device on postDeviceCommands:
boot.initrd.postDeviceCommands = ''
cd /sys/bus/platform/drivers/sdhci-arasan
mmctries=1000
[[ -e /dev/sda1 ]] && mmctries=20
for try in $(seq $mmctries); do
test -e fe330000.sdhci/mmc_host/mmc*/mmc*/block && found=1 && break
echo fe330000.sdhci > unbind || true
echo fe330000.sdhci > bind
sleep 1
done
test -n "$found" && echo "mmc appeared after $try attempts" | tee /dev/log
'';
}
2) Disable hs200 either in the device tree or bluntly in the kernel driver. The downside being eMMC accesses are at the slower mode by disabling hs200.
Reference solution (to be adapted): 0001-HACK-disables-hs400es-codepath.patch
15. Appendice
14.1. Serial connection
Use a device like SERIAL CONSOLE “Woodpecker”.
Schema:
Woodpecker <-> RockPro64
GND <-> Pin 6
RTX <-> Pin 8
TXD <-> Pin 10 (is optional: read-only mode if disconnected)
When booting RockPro64, Woodpecker's TX cable (white on picture) must be disconnected to avoid freezing. Reconnect after boot.
To stablish serial connection:
$ screen /dev/ttyUSB0 1500000
14.2. SPI
To erase SPI from:
1. U-boot console
=> sf probe
SF: Detected gd25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
=> sf erase 0 +1000000
SF: 16777216 @ 0x0 Erased: OK
2. NixOS
To skip SPI boot:
1. Physically jump pin 23 (SPI) to 25 (GND).
2. Remove jumper before OS loads to let OS recognize SPI device.
Install "mtd-utils" package.
# nix-env -i mtdutils
SPI is located at "/dev/mtd0". Check:
# mtdinfo /dev/mtd0
mtd0
Name: spi0.0
Type: nor
Eraseblock size: 4096 bytes, 4.0 KiB
Amount of eraseblocks: 4096 (16777216 bytes, 16.0 MiB)
Minimum input/output unit size: 1 byte
Sub-page size: 1 byte
Character device major/minor: 90:0
Bad blocks are allowed: false
Device is writable: true
To download:
# nanddump /dev/mtd0 -f mtd0.dump
To erase:
# flash_erase /dev/mtd0 0 0
To upload:
# nandwrite /dev/mtd0 some_firmware
14.3. To recover Nix-Store from corruption
Globally:
# sudo nix-store --verify --repair --check-contents
Individually:
# nix store verify /nix/store/ycaxlrjqgd07v5i0f3dmg7hz5621mfrc-perl5.32.0-Encode-Locale-1.05
# nix store repair /nix/store/ycaxlrjqgd07v5i0f3dmg7hz5621mfrc-perl5.32.0-Encode-Locale-1.05
SDCard is horrible media, easily corrupts data, is very slow. Use eMMC or attach a real disk for sanity.
Important: *USB-Sata dongles must have proper UASP support* in your Linux distribution. For Raspberry Pi 4, avoid dongles and go for Argon M.2 case, it has UASP support.
15. GitLab Runner Kubernetes operator
For running Gitlab Runner Kubernetes Operator jobs on aarch64, specify an Ubuntu (why?) aarch64 helper image:
[[runners]]
[runners.kubernetes]
image = "ubuntu"
helper_image = "gitlab/gitlab-runner-helper:ubuntu-arm64-latest"
helper_image_flavor = "ubuntu"
Otherwise errors as:
| ERROR: Job failed (system failure): prepare environment: waiting for pod running: pod status is failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
| WARNING: Failed to process runner builds=0 error=prepare environment: waiting for pod running: pod status is failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information executor=kubernetes
Just saved you a lot of time.
16. References
https://github.com/AshyIsMe/nixos-installer-rockpro64
https://nixos.wiki/wiki/NixOS_on_ARM/PINE64_ROCKPro64
https://www.linuxquestions.org/questions/linux-general-1/how-to-mount-img-file-882386/
https://rbf.dev/blog/2020/05/custom-nixos-build-for-raspberry-pis/
https://github.com/Robertof/nixos-docker-sd-image-builder/tree/master/packer
https://nixos.wiki/wiki/K3s