I'm using 3 x RockPro64 (80 USD) for bootstrapping a Kubernetes / K3s cluster, enabling CI builds for ARM apps and keeping a few lightweight services up.
Here's how I've installed NixOS on it:
You likely want and should use a newer image version like 20.09. But I'll be using an older version 20.03: 2020-08-31 19:09:22, which is outdated by the time you are reading it, I won't bother updating the download version because it is a moving target - latest is (usually) better, the important part is to understand the process, and then you get to decide which NixOS and U-boot version you want.
Also older NixOS versions were affected by a dmidecode bug which would cause RockPro64 to kernel panic.
Pick a version of NixOS 20.09, 20.03.
$ wget https://hydra.nixos.org/build/126272499/download/1/nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2 # (653.27 MiB)
$ sha256sum nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2
5613c806ddee0f62c7130780f31e40286296ae8b36a806c6f4f752a4f0c3a089
For .xz:
$ xz -d file_name.img.xz
For .zst:
$ zstd -d file_name.img.zst
For .bz2:
$ bzip2 -d nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2
Uncompressed "nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img" file (2,7G) is created.
$ export MYIMAGE=nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img
Pick a version of U-Boot.
From Hydra's nixpkg ubootRockPro64.aarch64-linux. Download build 126171011, in details tab, locate "Output store paths", valued as "/nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.07". To download U-Boot:
$ nix-env -i /nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.07
$ export UBOOT_PATH=/nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.07/
$ parted ${MYIMAGE} print
$ parted ${MYIMAGE} rm 1
$ parted ${MYIMAGE} print
$ dd if=${UBOOT_PATH}idbloader.img of=${MYIMAGE} conv=fsync,notrunc bs=512 seek=64
$ dd if=${UBOOT_PATH}u-boot.itb of=${MYIMAGE} conv=fsync,notrunc bs=512 seek=16384
The image is now bootable.
$ fdisk -l ${MYIMAGE}
Offset is 512 (Block size) * 77824 (Start), equals 39845888
# mount -o loop,offset=39845888 ${MYIMAGE} /mnt/myimage
Partition files are now accessible:
intj@nix-2700x:/mnt/myimage]$ tree -L 2
.
├── boot
│ ├── extlinux
│ └── nixos
├── lost+found [error opening dir]
├── nix
│ └── store
└── nix-path-registration
6 directories, 1 file
Find your SDCard device (or adapter). Be careful because on misidentification other device's data will be destroyed.
# lsblk -f
In my case it is "/dev/sde". Then, to write our image to it, replace 'of' with your SDCard device location:
# dd status=progress conv=fsync bs=1M if=${MYIMAGE} of=/dev/sde
# eject /dev/sde
Plug your SDCard to your RockPro64! And turn it on!
- RockPro64 is turned on when the white LED (close to USB ports) is turned on. If it is off, it has not powered on yet. [In that case I press reset button and usually works.]
- When powering off the RockPro64 board, remove power supply and wait a bit longer before powering it on again. Or it might not automatically power up when plugging back power and you'd have to press reset for powering it up.
- For 4k monitor/TV use HDMI 2.0 (or superior) cable.
IMPORTANT: `configuration.nix` which is used to configure NixOS does not exist yet. The installation image does not include a configuration.nix
On booting the image, you can generate a configuration.nix with `nixos-generate-config`.
To configure your system edit `/etc/nixos/configuration.nix` and to activate the new configuration run `nixos-rebuild switch` (as root).
Use the NixOS manual as reference to configure your system.
Interesting bits:
`configuration.nix` is only needed for building the system. Booting only requires the files in /nix and /boot.
NixOS image itself is generated from https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/cd-dvd/sd-image-aarch64.nix.
To receive IP from local DHCP server.
# k#/configuration.nix
networking.hostName = "k#"; # Define your hostname.
networking.interfaces.eth0.useDHCP = true;
Sample configuration for Ubiquiti EdgeRouter but any BGP router (Mikrotik, etc) might be configured as well (with different syntax):
# Basic network configuration (define IPs, gateway, bridge ports, etc.):
interfaces {
ethernet eth0 {
address 192.168.4.2/30 # Uplink
}
switch switch0 {
address 192.168.20.254/24
switch-port {
interface eth1 {}
interface eth2 {}
interface eth3 {}
interface eth4 {}
}
}
}
protocols {
static {
route 192.168.0.0/16 {
next-hop 192.168.4.1 {
description interno
}
}
}
}
# DHCP server configuration (to deliver IP address to hosts):
service {
dhcp-server {
disabled false
hostfile-update disable
shared-network-name dhcpd-bridge20 {
authoritative disable
subnet 192.168.20.0/24 {
default-router 192.168.20.254
dns-server 8.8.8.8
dns-server 8.8.4.4
lease 86400
start 192.168.20.10 {
stop 192.168.20.253
}
static-mapping rockpro-1 {
ip-address 192.168.20.1
mac-address 0e:19:50:50:70:6c
}
static-mapping rockpro-2 {
ip-address 192.168.20.2
mac-address 02:a6:8e:02:7d:c7
}
static-mapping rockpro-3 {
ip-address 192.168.20.3
mac-address 1e:67:92:6e:24:dd
}
}
}
}
}
### k1/configuration.nix ###
imports = [
(import ./k3s.nix { inherit config pkgs; })
];
### k1/k3s.nix ###
{ config, pkgs, ... }:
{
boot.kernelModules = [ "overlay" "br_netfilter" ]; # Needed for K3s?
boot.kernel.sysctl = {
"net.bridge-nf-call-ip6tables" = 1;
"net.bridge-nf-call-iptables" = 1;
"net.ipv4.ip_forward" = 1;
};
services.k3s = {
enable = true;
role = "server";
extraFlags = "--no-deploy servicelb --no-deploy traefik --bind-address 192.168.20.1 --tls-san 192.168.20.1 --node-ip=192.168.20.1 --node-external-ip=192.168.20.1 --write-kubeconfig-mode 644";
};
}
### k2/configuration.nix ###
imports = [
(import ./k3s.nix { inherit config pkgs; })
];
### k2/k3s.nix ###
{ config, pkgs, ... }:
{
boot.kernelModules = [ "overlay" "br_netfilter" ]; # Needed for K3s?
boot.kernel.sysctl = {
"net.bridge-nf-call-ip6tables" = 1;
"net.bridge-nf-call-iptables" = 1;
"net.ipv4.ip_forward" = 1;
};
services.k3s = {
enable = true;
role = "agent";
serverAddr = "https://192.168.20.1:6443";
token = "FILL-WITH-K3S-MASTER-TOKEN"; # Token located at k3s-master's '/var/lib/rancher/k3s/server/node-token'
extraFlags = "--node-ip=192.168.20.2 --node-external-ip=192.168.20.2"; # Update worker IPs accordingly.
};
}
Import kubeconfig file from Master:
$ scp k3s-master:/etc/rancher/k3s/k3s.yaml ~/.kube/
kubectl looks up configuration at "~/.kube/config" location. You can symlink k3s.yaml to config:
$ ln -s ~/.kube/k3s.yaml ~/.kube/config
To safeguard credentials set permission to no group/world access:
$ chmod 700 ~/.kube/k3s.yaml
Test access to cluster:
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k1 Ready master 15h v1.19.2+k3s-d38505b1-dirty 192.168.20.1 192.168.20.1 NixOS 21.03 (Okapi) 5.4.77 containerd://1.4.0-k3s1
k2 Ready 13h v1.19.2+k3s-d38505b1-dirty 192.168.20.2 192.168.20.2 NixOS 21.03 (Okapi) 5.4.77 containerd://1.4.0-k3s1
k3 Ready 12h v1.19.2+k3s-d38505b1-dirty 192.168.20.3 192.168.20.3 NixOS 21.03 (Okapi) 5.4.77 containerd://1.4.0-k3s1
MetalLB layer 2 (ARP) load balancing works only for failover, it does not load balance traffic. Avoid it. Use BGP instead. Be aware of MetalLB BGP limitations like traffic is splitted evenly between nodes even when running multiple instances of pods in same node.
Edge router sample configuration:
protocols {
bgp 64500 {
maximum-paths {
ibgp 32
}
neighbor 192.168.20.1 {
remote-as 64501
soft-reconfiguration {
inbound
}
}
neighbor 192.168.20.2 {
remote-as 54501
soft-reconfiguration {
inbound
}
}
neighbor 192.168.20.3 {
remote-as 54501
soft-reconfiguration {
inbound
}
}
parameters {
router-id 192.168.20.254
}
}
}
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
# On first install only
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
Sample MetalLB configuration for deliverying private IPs by default and public optionally using BGP:
# ./metallb-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
peers:
- peer-address: 192.168.20.254
peer-asn: 64500
my-asn: 64501
address-pools:
- name: default
protocol: bgp
auto-assign: true
avoid-buggy-ips: true
addresses:
- 192.168.128.0/19
- name: public
protocol: bgp
auto-assign: false
addresses:
- 189.113.69.96/29 # Change this to your public network!
- 179.191.211.24/29 # Change this to your public network!
Apply configuration:
$ kubectl apply -f ./metallb-config.yaml
https://github.com/AshyIsMe/nixos-installer-rockpro64
https://nixos.wiki/wiki/NixOS_on_ARM/PINE64_ROCKPro64
https://www.linuxquestions.org/questions/linux-general-1/how-to-mount-img-file-882386/
https://rbf.dev/blog/2020/05/custom-nixos-build-for-raspberry-pis/
https://github.com/Robertof/nixos-docker-sd-image-builder/tree/master/packer