Installing NixOS on RockPro64, configuring K3s + MetalLB BGP
2020-08-19

I'm using 5 x RockPro64 (80 USD) for bootstrapping a Kubernetes / K3s cluster, enabling CI builds for ARM apps and keeping a few lightweight services up.

RockPro64

Here's how I've installed NixOS on it:

1. NixOS ARM & U-Boot version

Ideally use a newer image version, like 20.09 or unstable. But I'll be using an older version 20.03: 2020-08-31 19:09:22 because updating version is a moving target - latest is (usually) better, the important part is to understand the process, and then you get to decide which NixOS and U-boot version you want.

2 Acquire NixOS for Aarch64 image

2.1. Option 1: Build from sources

Important: Requires an aarch64 host for building it!

$ git clone https://github.com/NixOS/nixpkgs.git; cd nixpkgs

$ nix-build nixos -I nixos-config=nixos/modules/installer/sd-card/sd-image-aarch64.nix -A config.system.build.sdImage

$ ls -l ./result/sd-image/nixos-sd-image-21.11pre-git-aarch64-linux.img.zst

2.1. Option 2: Download from Hydra

2.1.2.1 Pick NixOS version

20.09, 20.03, unstable.

$ wget https://hydra.nixos.org/build/126272499/download/1/nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2 # (653.27 MiB)

2.1.2.2 Check for corruption

$ sha256sum nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2
5613c806ddee0f62c7130780f31e40286296ae8b36a806c6f4f752a4f0c3a089

2.2. Decompress file

For .xz: $ xz -d file_name.img.xz

For .zst: $ zstd -d file_name.img.zst

For .bz2: $ bzip2 -d nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2

Uncompressed "nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img" file (2,7G) is created.

2.3. Export my image location (to save typing):

$ export MYIMAGE=nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img

3. Download U-Boot

Option 1: As a nix package using Nix (Nix should be already installed)

$ nix-env -i uboot-rockpro64-rk3399_defconfig

Option 2: From hydra: Pick U-Boot version

Troubleshooting:

Last time I tried U-boot latest was broken [error here]. If still broken, pin Uboot version as suggested bellow - it works.

From Hydra's nixpkg ubootRockPro64.aarch64-linux. Download build 126171011, in details tab, locate "Output store paths", valued as "/nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.07".

To download pinned version of U-Boot:

$ nix-env -i /nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.07

3.1. Export U-Boot location (to save typing):

$ export UBOOT_PATH=/nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.07/

4. Delete unused first partition from NixOS ARM image

4.1. List partitions

$ parted ${MYIMAGE} print

4.2. Delete partition 1

$ parted ${MYIMAGE} rm 1

4.2. Check if Partition 1 is gone.

$ parted ${MYIMAGE} print

5. Copy U-Boot to image


$ dd if=${UBOOT_PATH}idbloader.img of=${MYIMAGE} conv=fsync,notrunc bs=512 seek=64
$ dd if=${UBOOT_PATH}u-boot.itb of=${MYIMAGE} conv=fsync,notrunc bs=512 seek=16384
The image is now bootable.

6. Image overview [Optional]

6.1. Mount image.

$ fdisk -l ${MYIMAGE}

Offset is 512 (Block size) * 77824 (Start), equals 39845888

# mount -o loop,offset=39845888 ${MYIMAGE} /mnt/myimage

Partition files are now accessible:

intj@nix-2700x:/mnt/myimage]$ tree -L 2
    .
    ├── boot
    │   ├── extlinux
    │   └── nixos
    ├── lost+found [error opening dir]
    ├── nix
    │   └── store
    └── nix-path-registration
    
    6 directories, 1 file

7. Copy image to SDCard.

Find your SDCard device (or adapter). Be careful because on misidentification other device's data will be destroyed.

# lsblk -f

In my case it is "/dev/sde". Then, to write our image to it, replace 'of' with your SDCard device location:

# dd status=progress conv=fsync bs=1M if=${MYIMAGE} of=/dev/sde
# eject /dev/sde

8. Now test it

Plug your SDCard to your RockPro64! And turn it on!

8.1 Useful tips:

- RockPro64 is turned on when the white LED (close to USB ports) is turned on. If it is off, it has not powered on yet. [In that case I press reset button and usually works.]

- When powering off the RockPro64 board, remove power supply and wait a bit longer before powering it on again. Or it might not automatically power up when plugging back power and you'd have to press reset for powering it up.

- For 4k monitor/TV use HDMI 2.0 (or superior) cable.

9. NixOS Configuration

IMPORTANT: `configuration.nix` which is used to configure NixOS does not exist yet. The installation image does not include a configuration.nix

On booting the image, you can generate a configuration.nix with `nixos-generate-config`.

To configure your system edit `/etc/nixos/configuration.nix` and to activate the new configuration run `nixos-rebuild switch` (as root).

Use the NixOS manual as reference to configure your system.

Interesting bits:

`configuration.nix` is only needed for building the system. Booting only requires the files in /nix and /boot.

NixOS image itself is generated from https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/cd-dvd/sd-image-aarch64.nix.

10. Configure Network

10.1. At NixOS

To receive IP from local DHCP server.


    # k#/configuration.nix
    networking.hostName = "k#"; # Define your hostname.
    networking.interfaces.eth0.useDHCP = true;

10.2. At Router

Sample configuration for Ubiquiti EdgeRouter but any BGP router (Mikrotik, etc) might be configured as well (with different syntax):

# Basic network configuration (define IPs, gateway, bridge ports, etc.):


    interfaces {
        ethernet eth0 {
            address 192.168.4.2/30 # Uplink
        }
        switch switch0 {
            address 192.168.51.254/24
            switch-port {
                interface eth1 {}
                interface eth2 {}
                interface eth3 {}
                interface eth4 {}
                interface eth5 {}
            }
        }
    }
    protocols {
        static {
            route 192.168.0.0/16 {
                next-hop 192.168.4.1 {
                    description interno
                }
            }
        }
    }

# DHCP server configuration (to deliver IP address to hosts):


    service {
        dhcp-server {
            disabled false
            hostfile-update disable
            shared-network-name dhcpd-bridge51 {
                authoritative disable
                subnet 192.168.51.0/24 {
                    default-router 192.168.51.254
                    dns-server 8.8.8.8
                    dns-server 8.8.4.4
                    lease 86400
                    start 192.168.51.10 {
                        stop 192.168.51.253
                    }
                    static-mapping h1 {
                        ip-address 192.168.51.1
                        mac-address 0e:19:50:50:70:6c
                    }
                    static-mapping h2 {
                        ip-address 192.168.51.2
                        mac-address 02:a6:8e:02:7d:c7
                    }
                    # Add a static-mapping register for each node.
                }
            }
        }
    }

11. K3s set-up

11.1. Configure master node (192.168.51.1)


    {
        boot.kernelModules = [ "overlay" "br_netfilter" ]; # Needed for K3s?
        boot.kernel.sysctl = {
            "net.bridge-nf-call-ip6tables" = 1;
            "net.bridge-nf-call-iptables" = 1;
            "net.ipv4.ip_forward" = 1;
        };
        services.k3s = {
            enable = true;
            role = "server";
            extraFlags = "--no-deploy servicelb --no-deploy traefik --bind-address 192.168.51.1 --tls-san 192.168.51.1 --node-ip=192.168.51.1 --node-external-ip=192.168.51.1 --write-kubeconfig-mode 644";
        };
    }

11.2. Configure worker nodes (192.168.51.[2-5])


    {
        boot.kernelModules = [ "overlay" "br_netfilter" ]; # Needed for K3s?
        boot.kernel.sysctl = {
            "net.bridge-nf-call-ip6tables" = 1;
            "net.bridge-nf-call-iptables" = 1;
            "net.ipv4.ip_forward" = 1;
        };
        services.k3s = {
            enable = true;
            role = "agent";
            serverAddr = "https://192.168.51.1:6443";
            token = "FILL-WITH-K3S-MASTER-TOKEN"; # Token located at k3s-master's '/var/lib/rancher/k3s/server/node-token'
            extraFlags = "--node-ip=192.168.51.2 --node-external-ip=192.168.51.2"; # Update worker IPs accordingly.
        };
    }

12. Accessing K3s cluster

Import kubeconfig file from Master:

$ scp k3s-master:/etc/rancher/k3s/k3s.yaml ~/.kube/

kubectl looks up configuration at "~/.kube/config" location. You can symlink k3s.yaml to config:

$ ln -s ~/.kube/k3s.yaml ~/.kube/config

To safeguard credentials set permission to no group/world access:

$ chmod 700 ~/.kube/k3s.yaml

Test access to cluster:


    $ kubectl get nodes -o wide
    NAME   STATUS   ROLES                  AGE   VERSION                      INTERNAL-IP    EXTERNAL-IP    OS-IMAGE              KERNEL-VERSION   CONTAINER-RUNTIME
    k1     Ready    control-plane,master   1h    v1.20.5+k3s-355fff30-dirty   192.168.51.1   192.168.51.1   NixOS 21.05 (Okapi)   5.10.29          containerd://1.4.4-k3s1
    k2     Ready                     1h    v1.20.5+k3s-355fff30-dirty   192.168.51.2   192.168.51.2   NixOS 21.05 (Okapi)   5.10.29          containerd://1.4.4-k3s1
    k3     Ready                     1h    v1.20.5+k3s-355fff30-dirty   192.168.51.3   192.168.51.3   NixOS 21.05 (Okapi)   5.10.29          containerd://1.4.4-k3s1
    k4     Ready                     1h    v1.20.5+k3s-355fff30-dirty   192.168.51.4   192.168.51.4   NixOS 21.05 (Okapi)   5.10.29          containerd://1.4.4-k3s1
    k5     Ready                     1h    v1.20.5+k3s-355fff30-dirty   192.168.51.5   192.168.51.5   NixOS 21.05 (Okapi)   5.10.29          containerd://1.4.4-k3s1

13. MetalLB BGP

MetalLB layer 2 (ARP) load balancing works only for failover, it does not load balance traffic. Avoid it. Use BGP instead. Be aware of MetalLB BGP limitations like traffic is splitted evenly between nodes even when running multiple instances of pods in same node.

13.1. Configure router for BGP

Edge router sample configuration:


    protocols {
        bgp 64500 {
            parameters {
                router-id 192.168.51.254
            }
            maximum-paths {
                ibgp 32
            }
            neighbor 192.168.51.1 {
                remote-as 64501
                soft-reconfiguration {
                    inbound
                }
            }
            neighbor 192.168.51.2 {
                remote-as 64501
                soft-reconfiguration {
                    inbound
                }
            }
            # Add a neighbor register for each node. I won't repeat it.
        }
    }

13.2. Install MetalLB


    $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/namespace.yaml
    $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.6/manifests/metallb.yaml
    # On first install only
    $ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Sample MetalLB configuration for deliverying private IPs by default and public optionally using BGP:


    # ./metallb-config.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
        namespace: metallb-system
        name: config
    data:
        config: |
        peers:
        - peer-address: 192.168.51.254
            peer-asn: 64500
            my-asn: 64501
        address-pools:
        - name: default
            protocol: bgp
            auto-assign: true
            avoid-buggy-ips: true
            addresses:
            - 192.168.128.0/19
        - name: public
            protocol: bgp
            auto-assign: false
            addresses:
            - 189.113.69.96/29 # Change this to your public network!
            - 179.191.211.24/29 # Change this to your public network!

Apply configuration:

$ kubectl apply -f ./metallb-config.yaml

At Router, neighbors will show:


ubnt@ubnt-k3s:~$ show ip bgp summary
BGP router identifier 192.168.51.254, local AS number 64500
BGP table version is 1
0 BGP AS-PATH entries
0 BGP community entries
1  Configured ebgp ECMP multipath: Currently set at 1
32  Configured ibgp ECMP multipath: Currently set at 32

Neighbor                 V   AS   MsgRcv    MsgSen TblVer   InQ   OutQ    Up/Down   State/PfxRcd
192.168.51.1             4 64501    5          5       1      0      0  00:01:51               0
192.168.51.2             4 64501    5          5       1      0      0  00:01:51               0
192.168.51.3             4 64501    5          5       1      0      0  00:01:51               0
192.168.51.4             4 64501    5          5       1      0      0  00:01:51               0
192.168.51.5             4 64501    6          6       1      0      0  00:02:28               0

Total number of neighbors 5

Total number of Established sessions 5

Exposing a K8s Service will list as a connected route:


    ubnt@ubnt-k3s:~$ show ip bgp ipv4 unicast
BGP table version is 8, local router ID is 192.168.51.254
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, l - labeled
              S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network          Next Hop            Metric    LocPrf       Weight Path
*>  192.168.128.1/32 192.168.51.5         0                     0       64501 ?
*                    192.168.51.1         0                     0       64501 ?
*                    192.168.51.2         0                     0       64501 ?
*                    192.168.51.4         0                     0       64501 ?
*                    192.168.51.3         0                     0       64501 ?

Total number of prefixes 1

13.3. Optional: Reset K3s host


# systemctl stop k3s
# rm -rf /etc/rancher/k3s/
# rm -rf /etc/rancher/node/
# rm -rf /var/lib/rancher/k3s/
# nixos-rebuild switch

14. Troubleshooting

14.1. Won't boot with eMMC.

Serial output as in picture. In brief:

        
        mmc1: mmc_select_hs200 failed, error -110
        mmc1: error -110 whilsting MMC created

        mmc1: mmc_select_hs200 failed, error -110
        error -110 whilstd

        mmc1: mmc_select_hs200 failed, error -110
        mmc1: error -110 whilst initialising MMC card

        kbd_mode: KDSKBMODE: Inappropriate ioctl for device

        ...
        An error occurred in stage 1 of the boot process, which must mount the root filesystem on `/mnt-root' and then start stage 2. Press one of the following keys:

        r) to reboot immediately
        *) to ignore the error and continue
    

If you press 'r', the system will boot correctly. But you don't want to press 'r' every boot. :)

@samueldr suspects it's a marginal behaviour in the Linux mmc driver around hs200, hs400, access mode.

Workaround

# Workaround kindly offered by @samueldr

Either:

1) unbind/bind device on postDeviceCommands:

    boot.initrd.postDeviceCommands = ''
        cd /sys/bus/platform/drivers/sdhci-arasan
        mmctries=1000
        [[ -e /dev/sda1 ]] && mmctries=20
        for try in $(seq $mmctries); do
            test -e fe330000.sdhci/mmc_host/mmc*/mmc*/block && found=1 && break
            echo fe330000.sdhci > unbind || true
            echo fe330000.sdhci > bind
            sleep 1
        done
        test -n "$found" && echo "mmc appeared after $try attempts" | tee /dev/log
        '';
      }

Reference solutions: #1, #2

2) Disable hs200 either in the device tree or bluntly in the kernel driver. The downside being eMMC accesses are at the slower mode by disabling hs200.

Reference solution (to be adapted): 0001-HACK-disables-hs400es-codepath.patch

15. Appendice

14.1. Serial connection

Use a device like SERIAL CONSOLE “Woodpecker”.

Schema:
Woodpecker <-> RockPro64
GND <-> Pin 6
RTX <-> Pin 8
TXD <-> Pin 10 (is optional: read-only mode if disconnected)

When booting RockPro64, Woodpecker's TX cable (white on picture) must be disconnected to avoid freezing. Reconnect after boot.

To stablish serial connection:

$ screen /dev/ttyUSB0 1500000

14.2. SPI

To erase SPI from:

1. U-boot console

=> sf probe

SF: Detected gd25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB

=> sf erase 0 +1000000

SF: 16777216 @ 0x0 Erased: OK

2. NixOS

To skip SPI boot:
1. Physically jump pin 23 (SPI) to 25 (GND).
2. Remove jumper before OS loads to let OS recognize SPI device.

Install "mtd-utils" package.

# nix-env -i mtdutils

SPI is located at "/dev/mtd0". Check:

# mtdinfo /dev/mtd0
mtd0
Name:                           spi0.0
Type:                           nor
Eraseblock size:                4096 bytes, 4.0 KiB
Amount of eraseblocks:          4096 (16777216 bytes, 16.0 MiB)
Minimum input/output unit size: 1 byte
Sub-page size:                  1 byte
Character device major/minor:   90:0
Bad blocks are allowed:         false
Device is writable:             true

To download:

# nanddump /dev/mtd0 -f mtd0.dump

To erase:

# flash_erase /dev/mtd0 0 0

To upload:

# nandwrite /dev/mtd0 some_firmware

14.3. To recover Nix-Store from corruption

Globally:

# nix store verify --all

# nix store repair --all

Individually:

# nix store verify /nix/store/ycaxlrjqgd07v5i0f3dmg7hz5621mfrc-perl5.32.0-Encode-Locale-1.05

# nix store repair /nix/store/ycaxlrjqgd07v5i0f3dmg7hz5621mfrc-perl5.32.0-Encode-Locale-1.05

SDCard is horrible media, easily corrupts data, is very slow. Use eMMC or attach a real disk for sanity.
Important: *USB-Sata dongles must have proper UASP support* in your Linux distribution. For Raspberry Pi 4, avoid dongles and go for Argon M.2 case, it has UASP support.

15. GitLab Runner Kubernetes operator

For running Gitlab Runner Kubernetes Operator jobs on aarch64, specify an Ubuntu (why?) aarch64 helper image:


[[runners]]
    [runners.kubernetes]
        image = "ubuntu"
        helper_image = "gitlab/gitlab-runner-helper:ubuntu-arm64-latest"
        helper_image_flavor = "ubuntu"

Otherwise errors as:


    | ERROR: Job failed (system failure): prepare environment: waiting for pod running: pod status is failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

    | WARNING: Failed to process runner                   builds=0 error=prepare environment: waiting for pod running: pod status is failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information executor=kubernetes

Just saved you a lot of time.

16. References

https://github.com/AshyIsMe/nixos-installer-rockpro64

https://nixos.wiki/wiki/NixOS_on_ARM/PINE64_ROCKPro64

https://www.linuxquestions.org/questions/linux-general-1/how-to-mount-img-file-882386/

https://rbf.dev/blog/2020/05/custom-nixos-build-for-raspberry-pis/

https://github.com/Robertof/nixos-docker-sd-image-builder/tree/master/packer

https://nixos.wiki/wiki/K3s