Recently, I’ve been using the Compute Module 4 and a custom board with a Realtek RTL8111H PCIe Ethernet controller to develop a network router. However, the Raspberry Pi OS didn’t come with the required r8168 or r8169 drivers for the RTL8111. But fortunately, compiling and installing these drivers was relatively easy. Or at least until I discovered a neat trick: by adding arm_64bit=1 to the /boot/config.txt file, I could switch to a 64-bit kernel. This made the kernel 64-bit, but all user-space programs remained 32-bit. This was ideal for me, as all the routing work was retained in the kernel. Although a full 64-bit Raspberry Pi OS is still in development, I’m happy to stick with the 32-bit version for now.

•  PROJECTS
•  SOFTWARE
•  LIBRARIES
•  HOW-TO
•  MORE STUFF

Raspberry Pi – Compiling a Module for the 64-bit Kernel

 3 Jul 2021

Recently I’ve been working on a network router using the Compute Module 4 and a custom base board with a Realtek RTL8111H PCIe Ethernet controller. Raspberry Pi OS doesn’t come pre-installed with the r8168 or r8169 driver needed for the RTL8111, but getting it compiled and installed is pretty easy. Or at least until I found out that it’s possible switch to a 64-bit kernel by adding arm_64bit=1 to /boot/config.txt! The kernel is 64-bit, but all of the user space programs are still 32-bit. This is fine since all of the routing hard work stays within the kernel anyway. The full 64-bit Raspberry Pi OS is still in development so I wanted to stick with the 32-bit version for now.

UPDATE August 2022:
Raspberry Pi OS includes the r8169 driver since the 2022-01-28 release so now there’s no need to build the driver yourself.

I’ve never really touched Linux kernel stuff before. The first hurdle was that the raspbian repository doesn’t have a cross-compiler for 64-bit ARM (aarch64) or headers for the 64-bit kernel. However, the kernel source which includes the headers can be downloaded from the Raspberry Pi GitHub repo. After more googling I found that the best way of getting an aarch64 compiler would be download the ubuntu arm64 base from http://cdimage.ubuntu.com/ubuntu-base/releases/ and follow the steps described here and here.

I was still getting errors though. Mainly about a missing modules.lds file, but copying that file from the headers of the 32-bit kernel (/lib/modules/5.10.17-v7l+/build/scripts/) seemed to fix it. It’s just a small text file that doesn’t seem to be architecture specific.

The module finally compiled! But now I was getting Exec format errors when attempting to load it. modinfo r8168 seemed to say that everything was correct; compiled for my kernel version and as aarch64.

$modinfo r8168

[cut]

name: r8168

[cut]

vermagic: 5.10.17-v8 SMP preempt mod_unload modversions aarch64

[cut]

$sudo modprobe r8168 modprobe: ERROR: could not insert ‘r8168’: Exec format error

After more hours of futile attempts I finally came across this commit in the RPi kernel repo which mentions:

…you might have prepared the minimal setups for external modules by ‘make defconfig && make modules_preapre’…-*masahir0y*

Building an external module is exactly what I’ve been trying to do, so the make modules_prepare command looks promising! I ran the new command in the kernel source directory, recompiled the r8168 module and finally modprobe didn’t return an error! lsmod | grep r8168 shows that the driver has been loaded and a new network interface (eth1) appears in ip link show!

$ip link show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether dc:a6:32:ff:7d:f7 brd ff:ff:ff:ff:ff:ff
3: eth1: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 48:4d:7e:9f:e5:10 brd ff:ff:ff:ff:ff:ff

So, here’s the full set of commands needed to compile a 64-bit kernel module in 32-bit Raspberry Pi OS running a 64-bit kernel. There will probably be some warnings about missing symbols, but it doesn’t seem to be a problem.

OS: Raspberry Pi OS 10 (buster) Lite 2021-05-07 (32-bit)
Kernel: 5.10.17-v8+ (64-bit)
Module: r8168 v8.049.01

Grab the r8168 driver source from Realteks website.

mkdir ubuntu64
cd ubuntu64

wget http://cdimage.ubuntu.com/ubuntu-base/releases/21.04/release/ubuntu-base-21.04-base-arm64.tar.gz
tar -xf ubuntu-base-21.04-base-arm64.tar.gz

wget https://github.com/raspberrypi/linux/archive/refs/tags/raspberrypi-kernel_1.20210527-1.tar.gz
tar -xf raspberrypi-kernel_1.20210527-1.tar.gz

mv ../r8168-8.049.01.tar.bz2 .
tar -xf r8168-8.049.01.tar.bz2


cat /etc/resolv.conf >etc/resolv.conf
sudo mount -t tmpfs tmp tmp

sudo chroot .

apt update
apt install locales-all wget build-essential bc bison flex libssl-dev kmod

cd linux-raspberrypi-kernel_1.20210527-1/
export KCFLAGS="-march=armv8-a+crc -mtune=cortex-a72"
make -j4 bcm2711_defconfig
make -j4 modules_prepare

cd ../r8168-8.049.01
make -j4 -C ../linux-raspberrypi-kernel_1.20210527-1 M=$(pwd)/src modules

exit

sudo umount tmp


sudo mkdir /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/realtek/
sudo cp r8168-8.049.01/src/r8168.ko /lib/modules/$(uname -r)/kernel/drivers/net/ethernet/realtek/
sudo depmod $(uname -r)
sudo modprobe r8168