Lessons Learned: Raspberry Pi OS in-place upgrade

Background

I first started this journey because I wanted to setup a wireguard docker container on one of my pi’s. After doing the initial configuration the container wasn’t starting due to missing kernel headers. A few quick google searches later and it seems like upgrading Raspberry Pi OS from Buster to Bullseye will net me a newer kernel with the needed headers in place. Great! I’ve been meaning to tackle this at somepoint anyways.

Initial Steps Taken

I began by reading/following a guide by linuxuprising.com

1. Replace all instances of buster with bullseye in the apt sources

sudo sed -i 's/buster/bullseye/g' /etc/apt/sources.list
sudo sed -i 's/buster/bullseye/g' /etc/apt/sources.list.d/raspi.list
sudo sed -i 's/buster/bullseye/g' /etc/apt/sources.list.d/docker.list

2. Update apt sources

sudo apt update

3. Install gcc-8 to not run into dependency issues during upgrade

sudo apt install libgcc-8-dev gcc-8-base

4. Let’s get this upgrade started…

sudo apt full-upgrade

5. Remove any leftover unused packages

sudo apt autoremove

And for some reason this is around the time when I stopped following along and decided all was well and rebooted my pi.

Great Now it Won’t Boot

Hooking up a monitor confirmed we were now failing early in the boot process. More reading led to the realization that Bullseye changed the video driver used, which we’ll need to manually configure the system to use (this is done for you if doing a fresh install as recommended). Something I hadn’t even considered as I run all of my pi’s headless.

1. I removed the hard drive the pi boots from and pluged it into another computer
2. Modify the /boot/config.txt to enable the KMS driver. Start by commenting out any lines containing dtoverlay=vc4-fkms-v3d and adding dtoverlay=vc4-kms-v3d to the [all] section of the config:

sed -i 's/dtoverlay=vc4-fkms-v3d/#dtoverlay=vc4-fkms-v3d/g' <mount-path-of-drive>/boot/config.txt
sed -i 's/\[all\]/\[all\]\ndtoverlay=vc4-kms-v3d/' <mount-path-of-drive>/boot/config.txt

3. Unmount the drive from the other computer and reattach to the pi. Power it up, hope this is the end…

Compounding Issues

Well it made it farther this time. We boot up, but are met with a Failed to start DHCP Client Daemon error… meaning we have no network. Great at least we have a login prompt… Well turns out no combination of username/password work now. Fun.

Alright so we need to get a root shell

1. Remove the boot drive again, and plug it into another computer.
2. Modify the boot command to drop us into a root shell. Append init=/bin/sh to the end of /boot/cmdline.txt. For example:

console=serial0,115200 console=tty1 root=PARTUUID=16ae0856-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait init=/bin/sh

3. Reattach the boot drive to the pi and power it up
4. After booting and droping us into a root shell we need to mount the root partition as read-write:

mount -o remount, rw /

Its at this point I tired to change the password for my users account with a passwd <username> command, however I later found out this didn’t work. So onward with our DHCP issue.

5. Turns out the path to the dhcpcd binary has changed as well in our upgrade to bullseye. Thanks StackExchange:

sed -i 's|/usr/lib/dhcpcd5/dhcpcd|/usr/sbin/dhcpcd|g' /etc/systemd/system/dhcpcd.service.d/wait.conf

6. Now all I should need to do (famous last words) is sync the changes and boot:

sync
exec /sbin/init

Well if you hadn’t picked up on the foreshadowing by now, this wasn’t the end. The DHCP Client Daemon started this time, so progress! But I still couldn’t login for some reason.

The Final Hurdle

Luckily I hadn’t removed the init=/bin/sh line from the boot command, so I went through the whole reboot, mount the root partition as read-write song and dance again.

After trying the passwd command again, thinking maybe I made a typo; as well as adding a new user for good measure, I was starting to run out of ideas. My last ditch effort before calling it a night was running the raspi-config command and selecting the first option 1 Change User Password. And lo and behold this time I was able to login. Not completely sure why this worked and passwd didn’t but hey whatever, I’ll take the win and call it a night. That is after remembering to remove init=/bin/sh from /boot/cmdline.txt and rebooting. Thankfully everything came up as expected and I can now ssh in.