NixOS on Amlogic A311D (BPI-CM4)

While I'm waiting for my MNT Pocket Reform to ship I've decided to start working on porting NixOS to it. There are a couple of people mentioning NixOS on the MNT forum but I haven't seen anyone using NixOS on a Pocket Reform with the A311D module, yet.

To be able to do this I've bought a Banana Pi CM4 and the IO expansion baseboard, this setup is missing a lot of the hardware the MNT computer has - notably the LPC (system controller), real-time clock, the keyboard with its integrated menu, an M.2 slot, the screen, and probably more - but it's still better than to write everything blind.

Blue pcb on a white desk, it's the Banana Pi IO board and Compute Module 4.
Along the top edge are a USB C power input, micro SD card, empty SIM slot and HDMI connector, RJ45 ethernet and two empty USB A ports.
Right half of the IO board is covered by the SoM. It has an Amlogic A311D SoC, Rayson RAM, Samsung e-MMC, Realtek Ethernet controller and a Realtek Wifi and Bluetooth stamp module with two U.FL antena connectors next to it.
Left half of the IO board has a vacant mini-PCI Express connector.
Along the right edge is the typical Raspberry Pi 2.54mm spacing pin header.
Along the bottom edge are two flat-flex cable connectors labeled MIPI DSI, and MIPI CSI (for display and camera), two buttons, one Reset and one labeled "ADC2" which I don't know what does yet, and some indicator LEDs.
Along the left edge is 3 pin UART header which is connected to a cheap USB UART adapter connected to a black laptop that's mostly off screen.

MNT Debian image

As a sanity check that my setup works I've grabbed the latest official MNT Debian image from https://mnt.re/system-image (redirect to the latest pipeline). I flashed the image to an SD card with dd if=pocket-reform-system-a311d.img of=/dev/mmcblk0, inserted the SD card and it booted up. TTY over serial worked as expected but sway wouldn't launch because of the missing Pocket's DSI screen. To fix this I've had to modify the output configuration in ~/.sway/config. Remove the DSI screen and uncomment the HDMI example.

# output DSI-1 transform 270
# output DSI-1 scale 2
output HDMI-A-1 resolution 1920x1080 position 1920,0

When it boots the there is no video output, but I've managed to start sway by logging in as root (no password) on the serial tty, opening tmux and running sway. Then Ctrl-b c to create a second tmux window and:

# pgrep sway
1535
1544
# ls /run/user/0/sway-ipc.*
/run/user/0/sway-ipc.0.1535.sock
# export SWAYSOCK=/run/user/0/sway-ipc.0.1535.sock
# swaymsg output DSI-1 disable
# swaymsg output HDMI-A-1 enable

Adding those same directives into the sway config unfortunately didn't work for me, I have to repeat these steps manually to get a working display.

Desktop monitor showing the MNT Pocket setup wizard. The first screen contains a stylized drawing of the Pocket Reform and some hands typing on it. The drawing has black background with white outlines and colorful abstract shapes colored in with cyan, magenta and yellow solid colors.

Generic NixOS aarch64 SD card image

Even though it's pretty unlikely it'd work it's good to first check what will the "generic" image do. The latest image at the time was nixos-sd-image-24.11pre684053.9357f4f23713-aarch64-linux ¹. Flashing this image to the SD card with dd if=nixos-sd-image-24.11pre684053.9357f4f23713-aarch64-linux.img of=/dev/mmcblk0 and resetting doesn't work, the board stays in the same loop as without any SD card. (Whitespace added for clarity)

G12B:BL:6e7c85:2a3b91;FEAT:E0F83180:402000;POC:F;RCY:0;
       EMMC:0;READ:0;CHK:1F;READ:0;CHK:1F;READ:0;CHK:1F;SD?:0;SD:800;USB:8;
LOOP:1;EMMC:0;READ:0;CHK:1F;READ:0;CHK:1F;READ:0;CHK:1F;SD?:0;SD:800;USB:8;
LOOP:2;EMMC:0;READ:0;CHK:1F;READ:0;CHK:1F;READ:0;CHK:1F;SD?:0;SD:800;USB:8;
...

This is presumably because the generic NixOS image starts whith a whole lot of nothing instead of a bootloader. It has some useful suggestions in there though, unfortunately I must have misplaced my u-boot floppy disk.

00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
000001b0: 0000 0000 0000 0000 4e69 7821 0000 0005  ........Nix!....
000001c0: 0501 0bd7 1304 0040 0000 00f0 0000 80d7  .......@........
000001d0: 1404 830f 6285 0030 0100 182f 5e00 0000  ....b..0.../^...
000001e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000001f0: 0000 0000 0000 0000 0000 0000 0000 55aa  ..............U.
00000200: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
00800000: eb3c 906d 6b66 732e 6661 7400 0204 0400  .<.mkfs.fat.....
00800010: 0200 0200 f0f8 3c00 2000 0400 0000 0000  ......<. .......
00800020: 0000 0000 8000 294e 6978 2146 4952 4d57  ......)Nix!FIRMW
00800030: 4152 4520 2020 4641 5431 3620 2020 0e1f  ARE   FAT16   ..
00800040: be5b 7cac 22c0 740b 56b4 0ebb 0700 cd10  .[|.".t.V.......
00800050: 5eeb f032 e4cd 16cd 19eb fe54 6869 7320  ^..2.......This 
00800060: 6973 206e 6f74 2061 2062 6f6f 7461 626c  is not a bootabl
00800070: 6520 6469 736b 2e20 2050 6c65 6173 6520  e disk.  Please 
00800080: 696e 7365 7274 2061 2062 6f6f 7461 626c  insert a bootabl
00800090: 6520 666c 6f70 7079 2061 6e64 0d0a 7072  e floppy and..pr
008000a0: 6573 7320 616e 7920 6b65 7920 746f 2074  ess any key to t
008000b0: 7279 2061 6761 696e 202e 2e2e 200d 0a00  ry again ... ...
008000c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
008001f0: 0000 0000 0000 0000 0000 0000 0000 55aa  ..............U.
00800200: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
00800800: f8ff ffff ffff 0400 0500 0600 0700 0800  ................
00800810: 0900 0a00 0b00 0c00 0d00 0e00 0f00 1000  ................
00800820: 1100 1200 ffff 1400 1500 1600 1700 1800  ................
00800830: 1900 1a00 1b00 1c00 1d00 1e00 1f00 2000  .............. .
00800840: 2100 2200 2300 2400 ffff 2600 2700 2800  !.".#.$...&.'.(.
00800850: 2900 2a00 2b00 2c00 2d00 2e00 2f00 3000  ).*.+.,.-.../.0.
00800860: 3100 3200 3300 3400 3500 ffff 3700 3800  1.2.3.4.5...7.8.
00800870: 3900 3a00 3b00 3c00 3d00 3e00 3f00 4000  9.:.;.<.=.>.?.@.
00800880: 4100 4200 4300 4400 4500 ffff 4700 4800  A.B.C.D.E...G.H.
00800890: 4900 4a00 4b00 4c00 4d00 4e00 4f00 5000  I.J.K.L.M.N.O.P.
008008a0: 5100 5200 5300 5400 5500 5600 ffff 5800  Q.R.S.T.U.V...X.
008008b0: 5900 5a00 5b00 5c00 5d00 5e00 5f00 6000  Y.Z.[.\.].^._.`.
008008c0: 6100 6200 6300 6400 6500 6600 6700 ffff  a.b.c.d.e.f.g...
008008d0: 6900 6a00 6b00 6c00 6d00 6e00 6f00 7000  i.j.k.l.m.n.o.p.
008008e0: 7100 7200 7300 7400 7500 7600 7700 7800  q.r.s.t.u.v.w.x.
008008f0: 7900 7a00 7b00 7c00 7d00 7e00 7f00 8000  y.z.{.|.}.~.....

U-Boot

We could use MNT's build of the A311D u-boot image from their gitlab pipeline output, but while I was waiting for my A311D devkit to show up I've already attempted to nixify the uboot build. I know the outputs aren't identical but that could be a number of things, like a newer version of the compiler in NixOS compared to Debian Bookworm, so I'm still hopeful it worked and I want to try it first.

> nix build .#packages.x86_64-linux.ubootMntPocketReform
> file result/flash.bin
result/flash.bin: data

First thing I decided to flash it the same way it was done with the sd_fusing.sh script with Odroid u-boot image, seeking past the first 512 bytes to not disrupt the partition table.

> sudo dd if=result/flash.bin of=/dev/mmcblk0 conv=fsync,notrunc bs=512 seek=1
3077+1 records in
3077+1 records out
1575792 bytes (1,6 MB, 1,5 MiB) copied, 0,493206 s, 3,2 MB/s

It's not happy :< Still the same loop like we did nothing.

To check if it's our u-boot or not, let's try the same thing using MNT's u-boot image.

> unzip -l artifacts.zip 
Archive:  artifacts.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
  1578352  09-23-2024 22:10   meson-g12b-bananapi-cm4-mnt-pocket-reform-flash.bin
  1578352  09-23-2024 22:09   meson-g12b-bananapi-cm4-mnt-reform2-flash.bin
---------                     -------
  3156704                     2 files
> sudo dd if=meson-g12b-bananapi-cm4-mnt-pocket-reform-flash.bin of=/dev/mmcblk0 bs=512 skip=1
3081+1 records in
3081+1 records out
1577840 bytes (1,6 MB, 1,5 MiB) copied, 0,475595 s, 3,3 MB/s

Still not happy, it's not detecting a bootloader on the SD card. We must be flashing it wrong. Checking what our flash.bin looks like, it already starts with 512 bytes of zeros, so seeking another 512 bytes puts the MNTREFORMAMLBOOT tag at 0x400, but in the working image it starts 0x200.

00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
*
00000200: 4d4e 5452 4546 4f52 4d41 4d4c 424f 4f54  MNTREFORMAMLBOOT
00000210: 4041 4d4c f0ff 0000 4000 0101 0000 0000  @AML....@.......

And because the working image also starts with 512 bytes of zeros (oops!) we can just skip the seek.

> sudo dd if=result/flash.bin of=/dev/mmcblk0
3077+1 records in
3077+1 records out
1575792 bytes (1,6 MB, 1,5 MiB) copied, 0,455745 s, 3,5 MB/s

That helped! We can see u-boot output now. It's trying to do PXE boot though? Pressing keys until it drops us into the u-boot shell.

=> mmc list
sd@ffe03000: 2
sd@ffe05000: 0 (SD)
mmc@ffe07000: 1 (eMMC)
=> mmc dev 0
switch to partitions #0, OK
mmc0 is current device
=> mmc part
## Unknown partition table type 0

Huh.. did we just kill the partition table with zeros? I got carried away and forgot first 512 bytes are the partition table and not just zeros.. Let's repair it by copying it from the NixOS image we flashed to the SD card before.

> lsblk /dev/mmcblk0
NAME                MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
mmcblk0             179:0    0  29,7G  0 disk
> sudo dd if=nixos-sd-image-24.11pre684053.9357f4f23713-aarch64-linux.img of=/dev/mmcblk0 bs=512 count=1 
1+0 records in
1+0 records out
512 bytes copied, 0,00872231 s, 58,7 kB/s
> lsblk /dev/mmcblk0
NAME                MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
mmcblk0             179:0    0  29,7G  0 disk
├─mmcblk0p1         179:1    0    30M  0 part
└─mmcblk0p2         179:2    0   2,9G  0 part

And now we can start booting into linux.

U-Boot 2024.04 (Jan 01 1980 - 00:00:00 +0000)bpi-cm4-mnt-pocket-reform

Model: MNT Pocket Reform with BPI-CM4 Module
SoC:   Amlogic Meson G12B (A311D) Revision 29:b (10:2)
DRAM:  1 GiB (effective 3.8 GiB)
Core:  404 devices, 30 uclasses, devicetree: separate
MMC:   sd@ffe03000: 2, sd@ffe05000: 0, mmc@ffe07000: 1
Loading Environment from nowhere... OK
In:    usbkbd,serial
Out:   vidconsole,serial
Err:   vidconsole,serial
gpio: pin periphs-banks73 (gpio 88) value is 0
gpio: pin periphs-banks73 (gpio 88) value is 1
Net:   eth0: ethernet@ff3f0000
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:2...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
------------------------------------------------------------
1:	NixOS - Default
Enter choice: 1:	NixOS - Default
Retrieving file: /boot/extlinux/../nixos/kcfrpqk1nq08d56lb49h6ms1gwvn7g6d-linux-6.11-Image
Retrieving file: /boot/extlinux/../nixos/cpxmvya45xqblwvk60bpcv2h28nrfzr5-initrd-linux-6.11-initrd
append: init=/nix/store/wkkbg6gc0qnvzr4wjggzfq8yyg0y86hs-nixos-system-nixos-24.11pre684053.9357f4f23713/init console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 loglevel=7
Retrieving file: /boot/extlinux/../nixos/kcfrpqk1nq08d56lb49h6ms1gwvn7g6d-linux-6.11-dtbs/amlogic/meson-g12b-bananapi-cm4-mnt-pocket-reform.dtb
** File not found /boot/extlinux/../nixos/kcfrpqk1nq08d56lb49h6ms1gwvn7g6d-linux-6.11-dtbs/amlogic/meson-g12b-bananapi-cm4-mnt-pocket-reform.dtb **
Skipping fdtdir ../nixos/kcfrpqk1nq08d56lb49h6ms1gwvn7g6d-linux-6.11-dtbs for failure retrieving dts
Moving Image from 0x8080000 to 0x8200000, end=cc60000
## Flattened Device Tree blob at f0f27c30
   Booting using the fdt blob at 0xf0f27c30
Working FDT set to f0f27c30
   Loading Ramdisk to 3e969000, end 3ffff2b3 ... OK
   Loading Device Tree to 000000003e952000, end 000000003e96815f ... OK
Working FDT set to 3e952000

Starting kernel ...

Linux

Even without touching any files on the generic NixOS image we get some output. It has found the extlinux.conf config, which is nice because MNT uses some boot.scr partially-binary script which we don't have. I'm also a little surprised it knows to look for the Pocket Reform device tree without the boot script, is it baked into u-boot? It is! I should learn more about u-boot, that might be a fun future project.

So our problems are:

the Pocket Reform device tree is missing,
there is no output from the kernel, probably because the console= parameters are wrong, I think it should be tty=AML0,115200 like on the Odroid, as they're both Amlogic platforms.

I've already attempted to prepare a nix package of the MNT patched kernel, but I'm not comfortable with nixpkgs enough yet to create a custom SD card image. So instead of doing it the proper declarative nix way let's just copy the output files into /boot and edit the extlinux.conf that warns us to not edit it directly.

> nix build .#packages.x86_64-linux.linux_6_10_a311d_pocket_reform
> ls -l result/
total 58788
dr-xr-xr-x 1 root root       14 Jan  1  1970 dtbs
-r--r--r-- 1 root root 52761088 Jan  1  1970 Image
dr-xr-xr-x 1 root root       14 Jan  1  1970 lib
-r--r--r-- 1 root root  7430443 Jan  1  1970 System.map
> ls -l NIXOS_SD/boot/nixos/
total 95536
-r--r--r--  1 root root 23683763 Jan  1  1970 cpxmvya45xqblwvk60bpcv2h28nrfzr5-initrd-linux-6.11-initrd
dr-xr-xr-x 36 root root     4096 Jan  1  1970 kcfrpqk1nq08d56lb49h6ms1gwvn7g6d-linux-6.11-dtbs
-r--r--r--  1 root root 76829184 Jan  1  1970 kcfrpqk1nq08d56lb49h6ms1gwvn7g6d-linux-6.11-Image
> cat NIXOS_SD/boot/extlinux/extlinux.conf 
# Generated file, all changes will be lost on nixos-rebuild!

# Change this to e.g. nixos-42 to temporarily boot to an older configuration.
DEFAULT nixos-default

MENU TITLE ------------------------------------------------------------
TIMEOUT 50

LABEL nixos-default
  MENU LABEL NixOS - Default
  LINUX ../nixos/kcfrpqk1nq08d56lb49h6ms1gwvn7g6d-linux-6.11-Image
  INITRD ../nixos/cpxmvya45xqblwvk60bpcv2h28nrfzr5-initrd-linux-6.11-initrd
  APPEND init=/nix/store/wkkbg6gc0qnvzr4wjggzfq8yyg0y86hs-nixos-system-nixos-24.11pre684053.9357f4f23713/init console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 loglevel=7
  FDTDIR ../nixos/kcfrpqk1nq08d56lb49h6ms1gwvn7g6d-linux-6.11-dtbs

Our kernel package didn't output an initrd, generating it without a real system configuration might be complicated? Let's just try reusing the generic NixOS one first, that will save me some thinking if it happens to work.

> sudo cp -r result/dtbs NIXOS_SD/boot/nixos/testing-dtbs
> sudo cp result/Image NIXOS_SD/boot/nixos/testing-Image
> sudo cp -r result/lib/modules NIXOS_SD/lib/

I also have not figured out yet how NixOS tells the kernel where to load kernel modules from, after a bit of fruitless digging I tried just throwing them in /lib/modules like normal distributions do and it worked..

For the kernel cmdline we'll reference the working MNT image:

# cat /proc/cmdline 
ro no_console_suspend cryptomgr.notests loglevel=3  ro no_console_suspend console=ttyAML0,115200 pci=pcie_bus_perf libata.force=noncq nvme_core.default_ps_max_latency_us=0 console=tty1 fbcon=rotate:3  console=ttyAML0,115200 pci=pcie_bus_perf libata.force=noncq nvme_core.default_ps_max_latency_us=0 console=tty1

There is a lot of options and some of them duplicated.

console=tty1 - a virtual console, we don't need one yet²
console=ttyAML0,115200 - the Amlogic serial console device, as we suspected
cryptomgr.notests - disables crypto self-tests, for a minor boot speedup?³
fbcon=rotate:3 - rotates the linux console :3 the Pocket Reform builtin display needs this
libata.force=noncq - disables SATA NCQ (native command queueing), probably irrelevant? we don't use any SATA drives
loglevel=3 - kernel logging verbosity
no_console_suspend - disables suspending of consoles during suspend and hibernate operations. according to the docs this doesn't work reliably with all consoles but is known to work with serial consoles. this is going to be useful when debugging suspend or hibernate
nvme_core.default_ps_max_latency_us=0 - sets maximum power saving latency of PCIe devices to zero. might be useful to try when we get to test an NVMe drive⁴
pci=pcie_bus_perf - sets some PCIe performance settings, I don't want to poke this until I see the NVMe SSD is too slow or unreliable⁵
ro - mount root device read-only on boot. Not sure why it's here, maybe the official MNT image is meant as an installer only?

Let's try to change the extlinux.conf like this:

# Generated file, all changes will be lost on nixos-rebuild!

# Change this to e.g. nixos-42 to temporarily boot to an older configuration.
DEFAULT nixos-default

MENU TITLE ------------------------------------------------------------
TIMEOUT 50

LABEL nixos-default
  MENU LABEL NixOS - Default
  LINUX ../nixos/testing-Image
  INITRD ../nixos/cpxmvya45xqblwvk60bpcv2h28nrfzr5-initrd-linux-6.11-initrd
  APPEND init=/nix/store/wkkbg6gc0qnvzr4wjggzfq8yyg0y86hs-nixos-system-nixos-24.11pre684053.9357f4f23713/init console=ttyAML0,115200 console=tty1 loglevel=7
  FDTDIR ../nixos/testing-dtbs

There were a lot of scary looking messages in the log after this, but it booted and we got this screenshot out of it.

screenshot of a nix-shell with neofetch. the neofetch fields are:
OS: NixOS 24.11 Vicuna aarch64;
Host: MNT Pocket Reform with BPI-CM4 Module;
Kernel: 6.10.0;
Uptime: 1 minute;
Terminal: /dev/ttyAML0;
CPU: 6 cores at 1.8 Giga Hertz;
Memory: 4 Giga Bytes;

Proper NixOS config

Next step is to modify my nixos configuration to include this new board. Most of this time I've ~~wasted~~ spent trying to use the kernel I built as part of the reform-nixos-packages flake staring at errors like this one:

Screenshot of a terminal with a Nix build error message, it complains attribute 'override' is missing somewhere within nixpkgs module machinery, none of the trace is in my code.

We really really want to cross-compile the kernel because my desktop can manage it in several minutes and the board would be unbearably slow⁶. What I was attempting with the flake was something similar to what Lix does with their flake, however I'm not enough of a NixOS Witch yet to figure that one out in combination with the kernelPackages mechanism in nixpkgs. So I ended up with a mildly ugly hack, I'm re-importing nixpkgs specifically for the one package with system hardcoded to x86_64-linux and using the actual system as crossSystem.

Then to migrate the installation image to my new configuration I employed the same strategy as with the Odroid. We generate new partition names with uuidgen, configure the partition layout how we want it to look afterwards, generate random partition IDs with uuidgen, run nixos-rebuild boot --flake .#blueberry to build the configuration and put all files in place for next boot. And then, offline on a different computer, we backup the files, repartition the SD card and restore files from backup.

Booting from eMMC

The BPI-CM4 board has 16GB of onboard eMMC. I'm not expecting it to be particularly fast and we should avoid too much write wear to not kill it, but it'd be neat to install at least the bootloader on it to make the experience more like an x86 computer⁷ with builtin firmware.

I've found this community.mnt.re post to be a helpful reference, and this setup being not officially recommended yet made it sound even more interesting >:)

The relevant script is reform-tools:sbin/reform-flash-uboot which uses a platform specific configuration file. For me that's reform-tools:machines/MNT Pocket Reform with BPI-CM4 Module.conf. From these we learned there are special-purpose redundant partitions on the eMMC for flashing a bootloader to, they don't appear separate from within u-boot but in linux for the mmcblk1 internal eMMC there are also mmcblk1boot0 and mmcblk1boot1.

> lsblk /dev/mmcblk1boot?
NAME         MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
mmcblk1boot0 179:64   0   4M  1 disk 
mmcblk1boot1 179:96   0   4M  1 disk

These partitions are read-only by default, we can change this using sysfs file /sys/block/mmcblkXbootY/force_ro, writing 0 there allows writing and after flashing we can write 1 again.

Building the dd invocation the same way reform-flash-uboot does, using the config file variables UBOOT_OFFSET and FLASHBIN_OFFSET, leaves us with:

dd if=result/flash.bin of=/dev/mmcblk1boot0 bs=512 seek=1 skip=1

The SoC firmware bootloader expects u-boot at 0x200, as we learned by trial-and-error earlier. And because the start of mmcblk1boot0 is not a partition table like on the SD card, it's actually valid to simplify it. The whole process flashing from my reform-a311d-uboot flake looks like this:

> nix build .#packages.aarch64-linux.ubootMntPocketReform
> sha256sum result/flash.bin 
70cf7d4fb6acbfc1c426b084a1f73cdcff9d48ae81dd8c71bf39d1fa6e5ca63d  result/flash.bin
> echo 0 | sudo tee /sys/class/block/mmcblk1boot0/force_ro
0
> sudo dd if=result/flash.bin of=/dev/mmcblk1boot0 
3077+1 records in
3077+1 records out
1575792 bytes (1,6 MB, 1,5 MiB) copied, 0,19604 s, 8,0 MB/s
> echo 1 | sudo tee /sys/class/block/mmcblk1boot0/force_ro
1
> sudo head -c 1575792 /dev/mmcblk1boot0 | sha256sum
70cf7d4fb6acbfc1c426b084a1f73cdcff9d48ae81dd8c71bf39d1fa6e5ca63d  -

Now if we take out the SD card and reboot we get greeted by u-boot!

But we have 16 more gigabytes of free space on there, we could use it for /boot like some of the people in the community post said they do. This migration we can do online. We crate a single partition on /dev/mmcblk1 and format it as ext4. Then we update the /boot UUID in our NixOS config and run nixos-rebuild boot. And finally we mount the new boot partition somewhere and copy the contents of /boot to it.

Now booting without an SD card gets us all the way into the initrd, which waits for the / partition to appear and drops into a rescue shell when it times out.

<<< NixOS Stage 1 >>>

loading module btrfs...
loading module dm_mod...
running udev...
Starting systemd-udevd version 256.4
kbd_mode: KDSKBMODE: Inappropriate ioctl for device
starting device mapper and LVM...
Scanning for Btrfs filesystems
waiting for device /dev/disk/by-uuid/b8a8dcf6-e54d-4357-910b-088dc05c9ac1 to appear..................

This should make it easier to move to an NVMe / and setup full-disk encryption. Unfortunately I have to stick with using the SD card for a while longer because the BPI-CM4 devboard only has a mini-PCIe connector on it and I don't have an M.2 adapter yet.

WiFi

At this point I've noticed the wifi doesn't work, ip addr only shows an ethernet interface. Checking boot logs we see the driver fails to load firmware.

[   10.015715] rtw_8822cs mmc2:0001:1: failed to request firmware
[   10.019712] rtw_8822cs mmc2:0001:1: failed to request firmware
[   10.024689] rtw_8822cs mmc2:0001:1: failed to load firmware
[   10.030931] rtw_8822cs mmc2:0001:1: failed to setup chip efuse info
[   10.030939] rtw_8822cs mmc2:0001:1: failed to setup chip information
[   10.031107] rtw_8822cs mmc2:0001:1: probe with driver rtw_8822cs failed with error -22

On the MNT image there is a file /usr/lib/firmware/rtw88/rtw8822c_fw.bin which nix-locate found in rtw88-firmware and linux-firmware packages. We can install linux-firmware by setting hardware.enableRedistributableFirmware = true; in our NixOS config.

I don't have a Pocket Reform antenas, but I do have some spare parts thinkpads. So I borrowed the lid of an X200 for its antenas.

The blue Banana Pi board sitting inside a laptop lid backplate, gray and black antena cables from the lid are connected to its antena connectors. The ethernet cable is unplugged.

The signal strength is ok, in my unscientific testing of comparing a couple of devices laying next to each other. The A311D got average RSSI of about -57 dBm with the laptop lid laying flat on the desk, improving to -51 dBm if I pick it up and hold it upright. My laptop which is only slightly newer X230 and has a similar looking antena setup gets -49 dBm right next to it.

Unfortunately the wifi speed doesn't match the signal reception :( I expected it to not be great, there is a long thread about WiFi issues on the MNT Community forum, but this was still a shock to see how bad it is.

This is what iperf3 over wifi from the A311D looks like:

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  21.2 MBytes  17.8 Mbits/sec    0             sender
[  5]   0.00-10.04  sec  19.8 MBytes  16.5 Mbits/sec                  receiver

Compared to my laptop next to it:

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   242 MBytes   203 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   239 MBytes   200 Mbits/sec                  receiver

At least the ethernet speed is as expected?

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.10  sec  1.10 GBytes   933 Mbits/sec    0             sender
[  5]   0.00-10.11  sec  1.10 GBytes   933 Mbits/sec                  receiver

There is some interesting information in this post by Minute in the wifi issues thread.

If I do cat /sys/kernel/debug/mmc2/ios it really is at 25MHz and 3v3 signalling, while according to the thread it’s possible to run it at 200MHz 1v8.

We can check that quickly:

> sudo cat /sys/kernel/debug/mmc2/ios
clock:		25000000 Hz
actual clock:	25000000 Hz
vdd:		21 (3.3 ~ 3.4 V)
bus mode:	2 (push-pull)
chip select:	0 (don't care)
power mode:	2 (on)
bus width:	2 (4 bits)
timing spec:	0 (legacy)
signal voltage:	0 (3.30 V)
driver type:	0 (driver type B)

In some following posts in the same thread they include a device-tree patch and they said that the patch is included in an update.. huh, at the time of writing this is 14 days old, I might've picked up the wrong dts somewhere. The patch is included in main, and my flake.lock includes it, however it's in the meson-g12b-bananapi-cm4-mnt-reform2 directory and I've only included patches from meson-g12b-bananapi-cm4-mnt-pocket-reform! I'm not sure what other patches should I include if any, so for now let's try to add just the wifi patch.

With this patch applied we can verify it changed the mmc2 speed:

> sudo cat /sys/kernel/debug/mmc2/ios
clock:		100000000 Hz
actual clock:	99999999 Hz
vdd:		21 (3.3 ~ 3.4 V)
bus mode:	2 (push-pull)
chip select:	0 (don't care)
power mode:	2 (on)
bus width:	2 (4 bits)
timing spec:	6 (sd uhs SDR104)
signal voltage:	1 (1.80 V)
driver type:	0 (driver type B)

It did! Though only 100MHz, not 200MHz. And wifi performance?

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  80.0 MBytes  67.1 Mbits/sec    0             sender
[  5]   0.00-10.03  sec  77.9 MBytes  65.1 Mbits/sec                  receiver

Heck yea! It's still not amazing but a 3x improvement is great. Assuming no signal dropouts this is usable for me.

While I was waiting for the kernel to compile I tried to decipher more of the linux/build.sh file and I realized they might be applying all the patches and building only one linux kernel? Debian packaging scares me, I can't tell what's going on. We can try applying all the patches next and see what happens.

Another note for future me, a comment at :411 in this file reminded me that we get a bunch of warnings for non-existent Kconfig options and we might be not applying kernel patches before configuration?

Umm, here is a totally unrelated picture of the board in the dark, I liked how it looks winter holiday themed.

Grainy picture of the Banana Pi board, it's dark, the LEDs on the board itself, on the ethernet and on the UART adapter are green, blue, yellow and red, like winter holiday lights.

Other issues

Some more, yet unsolved issues I've run into.

Power management

Poweroff doesn't work, this might be actually related to me not running a real Pocket Reform because there poweroff is handled by the system controller which we're completely missing on the BPI devboard. Or it might be related to some still missing drivers.

[  OK  ] Reached target System Shutdown.
[  OK  ] Reached target Late Shutdown Services.
[  OK  ] Finished System Power Off.
[  OK  ] Reached target System Power Off.
[  621.641233] meson-drm ff900000.vpu: viu-hold-fifo-lines from device tree: 0
[  621.680009] meson-drm ff900000.vpu: viu-hold-fifo-lines from device tree: 0
[  621.718527] meson-drm ff900000.vpu: viu-hold-fifo-lines from device tree: 0
[  621.751788] meson-drm ff900000.vpu: viu-hold-fifo-lines from device tree: 0
...

The last message repeates forever and the system never turns off. However with the meson_drm module disabled the board resets on poweroff instead.

GPU/DRM

When the system boots up systemd-udevd and systemd-journal are loading the CPU significantly, the logs are getting spammed by these messages several times a second:

kernel: meson-dw-hdmi ff600000.hdmi-tx: Detected HDMI TX controller v2.01a with HDCP (meson_dw_hdmi_phy)
kernel: meson-dw-hdmi ff600000.hdmi-tx: registered DesignWare HDMI I2C bus driver
kernel: meson-drm ff900000.vpu: bound ff600000.hdmi-tx (ops meson_dw_hdmi_ops [meson_dw_hdmi])
kernel: meson-drm ff900000.vpu: viu-hold-fifo-lines from device tree: 0
kernel: meson-drm ff900000.vpu: CVBS Output connector not available

Running sudo rmmod meson_drm stops this spam with an additional logline:

kernel: platform sound: deferred probe pending: axg-sound-card: can't parse dai

But now I suspect we don't have any way to output video. Though it didn't seem to work even with the module enabled. This could maybe be caused by the device-tree assuming the builtin Pocket Reform display? Or it's the missing kernel patches.

After blocklisting the driver I've noticed the following in kernel logs, not sure if it was there before and just getting drowned out. This message sounds like it's complaining about the missing display - it names the specific panel Pocket Reform is supposed to have.

[   10.223245] panfrost ffe40000.gpu: error -ENODEV: _opp_set_regulators: no regulator (mali) found
[   10.229158] panel-jdi-lt070me05000 ffd07000.dsi.0: error -ENOENT: cannot get reset-gpios 0
[   10.234690] panel-jdi-lt070me05000 ffd07000.dsi.0: probe with driver panel-jdi-lt070me05000 failed with error -2

PCIe timeouts on bootup

When the kernel is starting up there is about a 2 second pause before systemd starts during which only this output is seen on the serial (with loglevel=4 on kernel cmdline).

Starting kernel ...

[    0.447302] meson-pcie fc000000.pcie: error: wait linkup timeout
[    0.489177] meson-pcie fc000000.pcie: error: wait linkup timeout
[    0.630593] meson-pcie fc000000.pcie: error: wait linkup timeout
[    0.771993] meson-pcie fc000000.pcie: error: wait linkup timeout
[    0.913385] meson-pcie fc000000.pcie: error: wait linkup timeout
[    1.054790] meson-pcie fc000000.pcie: error: wait linkup timeout
[    1.196281] meson-pcie fc000000.pcie: error: wait linkup timeout
[    1.337772] meson-pcie fc000000.pcie: error: wait linkup timeout
[    1.479301] meson-pcie fc000000.pcie: error: wait linkup timeout
[    1.619513] meson-pcie fc000000.pcie: error: wait linkup timeout
[    1.761067] meson-pcie fc000000.pcie: error: wait linkup timeout
[    1.905671] meson-pcie fc000000.pcie: error: wait linkup timeout

Uncomfy noises

Coil whine >.< When the board is powered up it makes a very uncomfortable noise, as far as I can tell it comes from the two larger coils on the SoM and not the smaller one on the IO board. I really hope this an issue with the IO board power supply not being very clean and the Pocket Reform will be quiet or I'll have to try covering the coils with potting epoxy or something.

Screenshot of a instantaneous spectrogram with a linear scale, from the android app Spectroid. It shows some activity down around 1 to 2 Kilo Hertz, and a hump of activity beyond 18 Kilo Hertz with a big peak at 20062 Hertz.

For now I'm wearing noise cancelling headphones when it's on which fortunately do a pretty good job with high pitched constant noises.

This link is probably going to be dead by the time I finish writing this article, you should be able to substitute with the latest succesful build of hydra.nixos.org/nixos.sd_image_new_kernel_no_zfs.aarch64-linux.

The kernel documentation for console= parameters is here Documentation/admin-guide/serial-console.rst. I think adding console=tty1 would not do anything as we don't have a working screen yet, but leaving it out shouldn't hurt.

I'm not sure why MNT enables this, might be some default from Debian. The kernel docs say basically nothing, but the kernel patch which introduces the option mentions the self-test takes "28ms on my laptop"... I'm sure this is fine to leave out.

⁴

This option is not listed in the kernel documentation but it's mentioned in a comment in drivers/nvme/host/core.c:2690, and in an error message in drivers/nvme/host/pci.c:1278. Both as debugging steps for problematic NVMe drives. I'm confused why setting maximum latency tolerance to zero helps with anything, I'd expect setting it to infinity would help. As far as I can trace the usage to drivers/nvme/host/core.c:4693 and the callee drivers/base/power/qos.c:900 zero is not a special value, negative values would disable it.

⁵

The kernel docs for pcie_bus_perf for anyone interested Documentation/admin-guide/kernel-parameters.txt:4544. I'll defer understanding these to future me.

⁶

Actually I'm curious, maybe it'll surprise me. I'll use the reform-nixos-pacakges repo for testing. They're not going to be running exactly the same workload, the x86 system is going to cross-compile and produce an aarch64 binary while the arm system will be producing a native binary, but that's close enough.

I've cleaned caches on both and then let the commands run until it started building linux-config, then I killed it and started again, to exclude network from the equation.

Desktop (AMD R7 3700X, 32GB memory, NVMe SSD with DRAM cache)

> time nix build .#packages.x86_64-linux.linux_6_10_a311d_pocket_reform --log-format multiline-with-logs
________________________________________________________
Executed in   28.29 mins    fish           external
   usr time    3.58 secs    0.00 micros    3.58 secs
   sys time    2.56 secs  811.00 micros    2.56 secs

BPI-CM4 (A311D, 4GB memory, no cooler, root on a V30 class microSD card)

> time nix build .#packages.aarch64-linux.linux_6_10_a311d_pocket_reform --log-format multiline-with-logs
________________________________________________________
Executed in  316.88 mins    fish           external
   usr time   11.29 secs    0.00 millis   11.29 secs
   sys time    7.63 secs   15.40 millis    7.61 secs

Ooof, that's rough, 11.2x slowdown. If I'm generous and say the 8 core machine has 16 cores thanks to SMT that's 4.2x slowdown when scaled by the core count difference. I'm curious how usable for big number crunching will the RK3588 board be, when I get my hands on that.

⁷

Speaking of making it feel more like an x86 system, I'm curious about trying Tow-Boot and ideally figuring out a way to access the u-boot shell or at least a boot menu without an external device. Either using the internal screen, which seems to be possible with u-boot if DSI support is added or trying to hack something together to the keyboard screen menu interface with u-boot, which could be really fun.