Installing NixOS on RockPro64, configuring K3s + MetalLB BGP
2020-08-19

I'm using 3 x RockPro64 (80 USD) for bootstrapping a Kubernetes / K3s cluster, enabling CI builds for ARM apps and keeping a few lightweight services up.

RockPro64

Here's how I've installed NixOS on it:

1. NixOS ARM & U-Boot version

You likely want and should use a newer image version like 20.09. But I'll be using an older version 20.03: 2020-08-31 19:09:22, which is outdated by the time you are reading it, I won't bother updating the download version because it is a moving target - latest is (usually) better, the important part is to understand the process, and then you get to decide which NixOS and U-boot version you want.
Also older NixOS versions were affected by a dmidecode bug which would cause RockPro64 to kernel panic.

2. Download NixOS ARM image.

Pick a version of NixOS 20.09, 20.03.

$ wget https://hydra.nixos.org/build/126272499/download/1/nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2 # (653.27 MiB)

2.1. Check for corruption:

$ sha256sum nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2
5613c806ddee0f62c7130780f31e40286296ae8b36a806c6f4f752a4f0c3a089

2.2. Decompress file:

For .xz:

$ xz -d file_name.img.xz

For .zst:

$ zstd -d file_name.img.zst

For .bz2:

$ bzip2 -d nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img.bz2
Uncompressed "nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img" file (2,7G) is created.

2.3. Export my image location (to save typing):

$ export MYIMAGE=nixos-sd-image-20.03.2868.ff6a070b4ef-aarch64-linux.img

3. Download U-Boot

Pick a version of U-Boot.

From Hydra's nixpkg ubootRockPro64.aarch64-linux. Download build 126171011, in details tab, locate "Output store paths", valued as "/nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.07". To download U-Boot:

$ nix-env -i /nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.07

3.1. Export U-Boot location (to save typing):

$ export UBOOT_PATH=/nix/store/k66wvh5h5vwdhx6n78nb50zpsn9ss2sg-uboot-rockpro64-rk3399_defconfig-2020.07/

4. Delete unused first partition from NixOS ARM image

4.1. List partitions

$ parted ${MYIMAGE} print

4.2. Delete partition 1

$ parted ${MYIMAGE} rm 1

4.2. Check if Partition 1 is gone.

$ parted ${MYIMAGE} print

5. Copy U-Boot to image


$ dd if=${UBOOT_PATH}idbloader.img of=${MYIMAGE} conv=fsync,notrunc bs=512 seek=64
$ dd if=${UBOOT_PATH}u-boot.itb of=${MYIMAGE} conv=fsync,notrunc bs=512 seek=16384
The image is now bootable.

6. Image overview [Optional]

6.1. Mount image.

$ fdisk -l ${MYIMAGE}

Offset is 512 (Block size) * 77824 (Start), equals 39845888

# mount -o loop,offset=39845888 ${MYIMAGE} /mnt/myimage

Partition files are now accessible:

intj@nix-2700x:/mnt/myimage]$ tree -L 2
    .
    ├── boot
    │   ├── extlinux
    │   └── nixos
    ├── lost+found [error opening dir]
    ├── nix
    │   └── store
    └── nix-path-registration
    
    6 directories, 1 file

7. Copy image to SDCard.

Find your SDCard device (or adapter). Be careful because on misidentification other device's data will be destroyed.

# lsblk -f

In my case it is "/dev/sde". Then, to write our image to it, replace 'of' with your SDCard device location:

# dd status=progress conv=fsync bs=1M if=${MYIMAGE} of=/dev/sde
# eject /dev/sde

8. Now test it

Plug your SDCard to your RockPro64! And turn it on!

8.1 Useful tips:

- RockPro64 is turned on when the white LED (close to USB ports) is turned on. If it is off, it has not powered on yet. [In that case I press reset button and usually works.]

- When powering off the RockPro64 board, remove power supply and wait a bit longer before powering it on again. Or it might not automatically power up when plugging back power and you'd have to press reset for powering it up.

- For 4k monitor/TV use HDMI 2.0 (or superior) cable.

9. NixOS Configuration

IMPORTANT: `configuration.nix` which is used to configure NixOS does not exist yet. The installation image does not include a configuration.nix

On booting the image, you can generate a configuration.nix with `nixos-generate-config`.

To configure your system edit `/etc/nixos/configuration.nix` and to activate the new configuration run `nixos-rebuild switch` (as root).

Use the NixOS manual as reference to configure your system.

Interesting bits:

`configuration.nix` is only needed for building the system. Booting only requires the files in /nix and /boot.

NixOS image itself is generated from https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/cd-dvd/sd-image-aarch64.nix.

10. Configure Network

10.1. At NixOS

To receive IP from local DHCP server.


    # k#/configuration.nix
    networking.hostName = "k#"; # Define your hostname.
    networking.interfaces.eth0.useDHCP = true;

10.2. At Router

Sample configuration for Ubiquiti EdgeRouter but any BGP router (Mikrotik, etc) might be configured as well (with different syntax):

# Basic network configuration (define IPs, gateway, bridge ports, etc.):


    interfaces {
        ethernet eth0 {
            address 192.168.4.2/30 # Uplink
        }
        switch switch0 {
            address 192.168.20.254/24
            switch-port {
                interface eth1 {}
                interface eth2 {}
                interface eth3 {}
                interface eth4 {}
            }
        }
    }
    protocols {
        static {
            route 192.168.0.0/16 {
                next-hop 192.168.4.1 {
                    description interno
                }
            }
        }
    }

# DHCP server configuration (to deliver IP address to hosts):


    service {
        dhcp-server {
            disabled false
            hostfile-update disable
            shared-network-name dhcpd-bridge20 {
                authoritative disable
                subnet 192.168.20.0/24 {
                    default-router 192.168.20.254
                    dns-server 8.8.8.8
                    dns-server 8.8.4.4
                    lease 86400
                    start 192.168.20.10 {
                        stop 192.168.20.253
                    }
                    static-mapping rockpro-1 {
                        ip-address 192.168.20.1
                        mac-address 0e:19:50:50:70:6c
                    }
                    static-mapping rockpro-2 {
                        ip-address 192.168.20.2
                        mac-address 02:a6:8e:02:7d:c7
                    }
                    static-mapping rockpro-3 {
                        ip-address 192.168.20.3
                        mac-address 1e:67:92:6e:24:dd
                    }
                }
            }
        }
    }

11. K3s set-up

11.1. Configure master node (192.168.20.1)


    ### k1/configuration.nix ###
    imports = [
        (import ./k3s.nix { inherit config pkgs; })
    ];

    ### k1/k3s.nix ###
    { config, pkgs, ... }:
    {
        boot.kernelModules = [ "overlay" "br_netfilter" ]; # Needed for K3s?
        boot.kernel.sysctl = {
            "net.bridge-nf-call-ip6tables" = 1;
            "net.bridge-nf-call-iptables" = 1;
            "net.ipv4.ip_forward" = 1;
        };
        services.k3s = {
            enable = true;
            role = "server";
            extraFlags = "--no-deploy servicelb --no-deploy traefik --bind-address 192.168.20.1 --tls-san 192.168.20.1 --node-ip=192.168.20.1 --node-external-ip=192.168.20.1 --write-kubeconfig-mode 644";
        };
    }

11.2. Configure worker nodes (192.168.20.2-#)


    ### k2/configuration.nix ###
    imports = [
        (import ./k3s.nix { inherit config pkgs; })
    ];

    ### k2/k3s.nix ###
    { config, pkgs, ... }:
    {
        boot.kernelModules = [ "overlay" "br_netfilter" ]; # Needed for K3s?
        boot.kernel.sysctl = {
            "net.bridge-nf-call-ip6tables" = 1;
            "net.bridge-nf-call-iptables" = 1;
            "net.ipv4.ip_forward" = 1;
        };
        services.k3s = {
            enable = true;
            role = "agent";
            serverAddr = "https://192.168.20.1:6443";
            token = "FILL-WITH-K3S-MASTER-TOKEN"; # Token located at k3s-master's '/var/lib/rancher/k3s/server/node-token'
            extraFlags = "--node-ip=192.168.20.2 --node-external-ip=192.168.20.2"; # Update worker IPs accordingly.
        };
    }

12. Accessing K3s cluster

Import kubeconfig file from Master:

$ scp k3s-master:/etc/rancher/k3s/k3s.yaml ~/.kube/

kubectl looks up configuration at "~/.kube/config" location. You can symlink k3s.yaml to config:

$ ln -s ~/.kube/k3s.yaml ~/.kube/config

To safeguard credentials set permission to no group/world access:

$ chmod 700 ~/.kube/k3s.yaml

Test access to cluster:


    $ kubectl get nodes -o wide
    NAME   STATUS   ROLES    AGE   VERSION                      INTERNAL-IP    EXTERNAL-IP    OS-IMAGE              KERNEL-VERSION   CONTAINER-RUNTIME
    k1     Ready    master   15h   v1.19.2+k3s-d38505b1-dirty   192.168.20.1   192.168.20.1   NixOS 21.03 (Okapi)   5.4.77           containerd://1.4.0-k3s1
    k2     Ready       13h   v1.19.2+k3s-d38505b1-dirty   192.168.20.2   192.168.20.2   NixOS 21.03 (Okapi)   5.4.77           containerd://1.4.0-k3s1
    k3     Ready       12h   v1.19.2+k3s-d38505b1-dirty   192.168.20.3   192.168.20.3   NixOS 21.03 (Okapi)   5.4.77           containerd://1.4.0-k3s1

13. MetalLB BGP

MetalLB layer 2 (ARP) load balancing works only for failover, it does not load balance traffic. Avoid it. Use BGP instead. Be aware of MetalLB BGP limitations like traffic is splitted evenly between nodes even when running multiple instances of pods in same node.

13.1. Configure router for BGP

Edge router sample configuration:


    protocols {
        bgp 64500 {
            maximum-paths {
                ibgp 32
            }
            neighbor 192.168.20.1 {
                remote-as 64501
                soft-reconfiguration {
                    inbound
                }
            }
            neighbor 192.168.20.2 {
                remote-as 54501
                soft-reconfiguration {
                    inbound
                }
            }
            neighbor 192.168.20.3 {
                remote-as 54501
                soft-reconfiguration {
                    inbound
                }
            }
            parameters {
                router-id 192.168.20.254
            }
        }
    }

13.2. Install MetalLB


    $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
    $ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
    # On first install only
    $ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Sample MetalLB configuration for deliverying private IPs by default and public optionally using BGP:


    # ./metallb-config.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
        namespace: metallb-system
        name: config
    data:
        config: |
        peers:
        - peer-address: 192.168.20.254
            peer-asn: 64500
            my-asn: 64501
        address-pools:
        - name: default
            protocol: bgp
            auto-assign: true
            avoid-buggy-ips: true
            addresses:
            - 192.168.128.0/19
        - name: public
            protocol: bgp
            auto-assign: false
            addresses:
            - 189.113.69.96/29 # Change this to your public network!
            - 179.191.211.24/29 # Change this to your public network!

Apply configuration:

$ kubectl apply -f ./metallb-config.yaml

14. References

https://github.com/AshyIsMe/nixos-installer-rockpro64

https://nixos.wiki/wiki/NixOS_on_ARM/PINE64_ROCKPro64

https://www.linuxquestions.org/questions/linux-general-1/how-to-mount-img-file-882386/

https://rbf.dev/blog/2020/05/custom-nixos-build-for-raspberry-pis/

https://github.com/Robertof/nixos-docker-sd-image-builder/tree/master/packer