A Sunday Morning Boot Problem

I woke up today to my computer not booting into my arch installation, but into memtest86+.A few months ago I also had problems with booting when I flashed a new BIOS¹ that turned out to be a beta version (thanks MSI!) and not working. At the time I removed GRUB and decided to use EFISTUB instead, since I don't need anything fancy for my booting; I only have one disk from which I boot.

After having changed to EFISTUB I had some problems the first times I upgraded my Linux version; when a new version is installed you build two important files, vmlinuz-linux and initramfs-linux.img, which, as far as I can tell, are the kernel itself and the initial data you want to be in RAM. So, when you update linux you'll get a new vmlinuz-linux with that new version:

/h/mht$ file /boot/EFI/arch/vmlinuz-linux
/boot/EFI/arch/vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 5.7.2-arch1-1 (linux@archlinux) #1 SMP PREEMPT Wed, 10 Jun 2020 20:36:24 +0000, RO-rootFS, swap_dev 0x7, Normal VGA

The problem I got was that the new generated files were put in /boot, but my EFI partition, which should either contain (or know the location of) the files above, was mounted to /boot/efi, and so when I tried to boot there was a mismatch between the linux image loaded, which was the old version in /boot/efi and the new version, which was installed to my system at /.

The solution was to make a systemd service thingy that would run whenever /boot/initramfs-linux-fallback.img changed and copy the three files into /boot/efi/EFI/arch. This worked, and all was well.

That is, until this morning.

I don't know why it stopped working, but it suddenly did, and my system refused to boot. In the boot options menu from my motherboard I still had the correct boot entry, but upon selection it would flash black and a message along the lines of

The file '\EIF\arch\vmlinuz-linux' could not be found.

would flash for about a frame.

I flashed a USB drive with the June arch .iso on it, and looked around in the UEFI shell, which I got kind of familiar with from the last time I messed around with these things. I was able to find the linux image on the EFI partition, and boot it with the right kernel parameters saying which block device is to be mounted as root (which involves typing in a long PARTUUID) and where the initramfs file is. Luckily it all worked, which kind of rules out hardware failure.

I went back and forth a bit, trying to edit the boot entries with bcfg in the UEFI shell or with efibootmgr after having arch-chrooted into my disk from the live flash drive. Nothing seemed to work; in fact, nothing even seemed wrong about the boot entry that I had from before.

I didn't really know what to try out next, so I tried to trim some of the paths to the files on the EFI partition from being full to just the filename. This did not work. Then, looking through the EFI system partition page on the Arch wiki I noticed the following:

/efi is a replacement[6] for the previously popular (and possibly still used by other Linux distributions) ESP mountpoint /boot/efi.

Alright, maybe this is an issue for some reason. I changed it and updated the systemd scripts, but before I got the chance to test it I read a little bit more on the wiki:

mount ESP to /boot. This is the preferred method when directly booting a EFISTUB kernel from UEFI.

Oh okay, I guess I'll just mount it to /boot then. This even means I don't need the systemd scripts anymore since this is the default place in which to put the files.

> mkdir boot
> cp /boot/* boot
> du -h boot
60M
> rm -r /boot/*
> rm -r /efi
> mount /dev/nvme0n1p1 /boot
> mv -r boot/ /boot/
> kak /etc/fstab # update mount point
> rm -r boot

Restart, and now it all works. This time I also tried the two boot entries with relative and absolute paths, and they both worked. Note that I didn't have to change the boot entries since they contain the file paths within the ESP partition, and I didn't change anything on the partition itself, only where in the root partition the ESP partition would be mounted. This is what's strange about it all to me.

The whole ordeal took about 3 hours, from starting the download of the arch ISO to the final working boot.

The day after writing this I stumbled upon a post on /r/archlinux. It turns out that the mount point of my EFI partition wasn't the problem after all, but the path containing forward slashes instead of backslashes were. Apparently, the parsing code was rewritten, and it just so happens that it accidently worked before.

Looking back, I can't really make sense of this, since I thought I'd tried to use the proper slashes after having read that the backslashes are the way EFI wants it. This is especially strange since I tried once to not have the full path but only the filenames (perhaps I started with a /?). Oh well.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License