Contents: boot-once testing hardware-less testing as a goal sequence of operations

Here are some notes about provisioning the potato board.

See notes about general provisioning strategy for ttc at: Provisioning Notes

The potato board is an ARM64 board I got from Kevin Hilman in September 2019. The image for it is an arm64 sdcard image, with the following:

Booting via grub, with the following sdcard partitions:

• vfat efi partition
• btrfs root partition with 3 sub-volumes
• swap partition

I intended to use grub to do a 'hardware-less' test environment on the board, but this proved difficult.

I intend to keep a working bootloader and linux image on the sdcard, and use that to boot test kernels.

After making a test kernel entry for grub, I found that I could not use grub's 'boot-once' feature, because grub doesn't know how to write to the btrfs filesystem.

Therefore, I migrated the filesystem to be as follows:

Booting via grub, with the following sdcard partitions:

• efi partition
• ext4 boot partition
• btrfs root partition with 3 sub-volumes
• swap partition

This was harder then I thought it would be, and involved a lot of manual steps.

Here are the main steps I had to do, after cloning the original sdcard:

• repartition the drive used gparted
  • resize the btrfs partition to make room for a new 300M partition at the front of the drive
    • this was possibly harder than needed, because I might have been able to shrink the partition to put the new partition at the end with less time. Or I could have sacrificed part of the swap partition. But I wasn't sure that grub could read from the end of the drive (that's probably an old LILO issue)
  • add a new 300M linux partition
  • label the new partition 'boot'
• reorder the partitions using fdisk
• reinstall arm64 grub, from ubuntu x86_64

After mounting /dev/sda partitions onto

• /mnt/sdcard = subvolume @ of /dev/sda3
• /mnt/sdcard/boot = /dev/sda2
• /mnt/sdcard/boot/efi = /dev/sda1

I ran the following:

   $ grub-install --directory=/mnt/sdcard/usr/lib/grub/arm64-efi \
       --boot-directory=/mnt/sdcard/boot \
       --bootloader-id=GRUB \
       --efi-directory=/mnt/sdcard/boot/efi \
       --target=arm64-uefi \
       /dev/sda

From here, I performed some operations while on the system itself.

I had renamed the '/boot' directory in the btrfs filesystem to '/bootold'. I temporarily continued to use this filesystem to boot the kernel, so I could continue updating the grub configuration.

I booted into the grub menu, and edited the linux kernel line to load the image from '/@/bootold/vmlinuz-4.15.55++' and initrd from the same directory. (eliminating the old reference to subvolume '/@/boot')

Then, after I booted, I found the UUID for mmcblk1p2 (ext4 boot partition) and added an entry to /etc/fstab to mount the new boot partition in /boot. You can use 'blkid' or 'lsblk -f' for to get the UUIDs for partitions.

With the correct boot partition loaded, I could now run 'update-grub' to update the grub configuration with new data.

I had to manually change the 'test kernel' grub configuration for this. This needs to be a script.

boot-once testing [edit section]

Boot-once testing is a configuration for provisioning the board that uses a 'safe' kernel and a test kernel. The test kernel is written to flash (or SDcard or disk) using the safe kernel, and the user can always boot the board using the safe kernel if something goes wrong with the test kernel (it fails to boot).

The reason for using this configuration is that it allows for recovering from failed test kernels, without requiring any additional hardware.

Boot-once testing requires that the bootloader be able to distinguish when it should boot the safe kernel and the test kernel. It needs to use some piece of information to do this. If the bootloader has access to the network, then it could read a value from some external device.

grub can read a file called /boot/grub/grubenv to tell give it information for the next boot. Note that this file can be written to by a safe kernel, but not necessarily by a test kernel. Therefore, for the bootloader to be able to tell itself to boot a kernel only once, it needs to have the bootloader write whether it succeeded or not into the filesystem. Grub cannot write to grubenv in the btrfs filesystem, but it can in an ext4 filesystem.

The tool 'grub-set-default' was used to write "Test kernel" as the default boot kernel.

I always try to maintain a "safe" kernel as the 0th entry in the grub menu, as that will be what grub falls back to as it's default, grubenv doesn't specify a default, or if the previous default fails to boot (ie, the test kernel fails to boot)

(question: Does the test kernel have to write to grubenv, or does grub record the failcount automatically? answer: I don't remember)

hardware-less testing as a goal [edit section]

Most test labs use a serial connection to the board to control the bootloader, to select the kernel under test or to provide command line parameters or ram addresses. However, this requires that the board expose a serial port, and that the user install a serial cable from the management host to the board. Many products either do not expose a serial port at all, or the port is only accessible with great difficulty (like requiring soldering to the back of the board, after the product is taken apart).

The goal of Fuego is to support hardware-less testing, so great lengths have been gone to to avoid having to require a serial cable or additional hardware by the user.

sequence of operations [edit section]

To provision pot1 (overview):

• build the kernel, modules, initrd
• boot into safe kernel
• copy kernel, modules, initrd to pot1:/boot/test-*
• update-grub
• boot into test kernel

Details:

• boot into test:
  • (in safe kernel): grub-set-default "Test kernel"
    • this sets: saved_entry="Test kernel" in /boot/grub/grubenv

Notes: The test kernel has "recordfail" as an action.