Building (Debian) kernel optimized for RPi Zero (W) and 1

[Diederik de Haas]

Hi,

Debian provides a kernel for Raspberry Pi Zero (W) and 1, but that targets
the armel architecture. I want to compile a/the Debian kernel that does use the
HW capabilities of the RPi 0/1, similarly to how raspbian.org recompiles
the Debian packages. Except AFAIK the Debian kernel.

I know that Debian won't change their RPi kernel for armel, but this is for
private use. But I don't know what the best way to go about that is.

I found the following link wrt building a cross compiler
https://solarianprogrammer.com/2018/05/06/building-gcc-cross-compiler-raspberry-pi/
which in turn references
https://preshing.com/20141119/how-to-build-a-gcc-cross-compiler/
and that does look promising.

But before diving full into it, I'd like to know whether this is the proper
approach or not. And if someone has good/better links, I'd appreciate it if
you'd share them :-)

While I (technically) could setup a system to compile natively, I want to use
cross compilation (on amd64 PC). I think it would otherwise take days.
But knowing how to compile natively would be nice too.
I do know how to compile Debian's kernels, both natively and cross building.

Cheers,
Diederik

[gene heskett]

Post by Diederik de Haas
Hi,
Debian provides a kernel for Raspberry Pi Zero (W) and 1, but that targets
the armel architecture. I want to compile a/the Debian kernel that does use the
HW capabilities of the RPi 0/1, similarly to how raspbian.org recompiles
the Debian packages. Except AFAIK the Debian kernel.
I know that Debian won't change their RPi kernel for armel, but this is for
private use. But I don't know what the best way to go about that is.
I found the following link wrt building a cross compiler
https://solarianprogrammer.com/2018/05/06/building-gcc-cross-compiler-raspberry-pi/
which in turn references
https://preshing.com/20141119/how-to-build-a-gcc-cross-compiler/
and that does look promising.
But before diving full into it, I'd like to know whether this is the proper
approach or not. And if someone has good/better links, I'd appreciate it if
you'd share them :-)
While I (technically) could setup a system to compile natively, I want to use
cross compilation (on amd64 PC). I think it would otherwise take days.

Probably an over estimate unless you don't have any SSD hardware for
workspace.
 I do, 240Gigs on a startech usb to sata adaptor  and I have built an
armhf version
of 4.19 in under an hour on an rpi4. Working on a u-sd card,
you'll be abusing the card and quite a few hours. I would not even try
it on less than a 64G u-sd.
You'll surely kill an 8G u-sd card by writing it to death..

Post by Diederik de Haas
But knowing how to compile natively would be nice too.
I do know how to compile Debian's kernels, both natively and cross building.
Cheers,
Diederik

Cheers, Gene Heskett.

[Punit Agrawal]

Post by Diederik de Haas
Hi,
Debian provides a kernel for Raspberry Pi Zero (W) and 1, but that targets
the armel architecture. I want to compile a/the Debian kernel that does use the
HW capabilities of the RPi 0/1, similarly to how raspbian.org recompiles
the Debian packages. Except AFAIK the Debian kernel.
I know that Debian won't change their RPi kernel for armel, but this is for
private use. But I don't know what the best way to go about that is.
I found the following link wrt building a cross compiler
https://solarianprogrammer.com/2018/05/06/building-gcc-cross-compiler-raspberry-pi/
which in turn references
https://preshing.com/20141119/how-to-build-a-gcc-cross-compiler/
and that does look promising.

I am probably missing some context but I wanted to point to the armel
cross compiler in Debian - gcc-arm-linux-gnueabi. Is there some reason
to build the cross-compiler yourself?

[...]

[Diederik de Haas]

Post by Punit Agrawal
I am probably missing some context but I wanted to point to the armel
cross compiler in Debian - gcc-arm-linux-gnueabi. Is there some reason
to build the cross-compiler yourself?

I don't want to build for armel, but for raspbian.org's variant of armhf,
which is not the same as Debian's armhf.
The RPi 0/1 fall somewhere in the middle of Debian's armel and armhf and
that's the whole reason raspbian.org was created.

But it can be that I'm looking at this problem all wrong and/or have some
tunnel vision towards building a cross compiler.
That is an important reason for making this ML thread.

Cheers,
Diederik

[Punit Agrawal]

Post by Diederik de Haas

Post by Punit Agrawal
I am probably missing some context but I wanted to point to the armel
cross compiler in Debian - gcc-arm-linux-gnueabi. Is there some reason
to build the cross-compiler yourself?

I don't want to build for armel, but for raspbian.org's variant of armhf,
which is not the same as Debian's armhf.
The RPi 0/1 fall somewhere in the middle of Debian's armel and armhf and
that's the whole reason raspbian.org was created.

Thanks for the context.

Post by Diederik de Haas
But it can be that I'm looking at this problem all wrong and/or have some
tunnel vision towards building a cross compiler.
That is an important reason for making this ML thread.

IIUC, for the kernel, either compiler should be fine. The
documentation[0] for compiling the RPi Zero kernel seems to bear this
out - it even uses the "CROSS_COMPILE=arm-linux-gnueabihf-". If you've
got access to the hardware it should be easy to test before kicking off
the Debian kernel package build.

Hope that helps.

Regards,
Punit

[0] https://www.raspberrypi.com/documentation/computers/linux_kernel.html#cross-compiling-the-kernel

[Marcin Juszkiewicz]

Post by Punit Agrawal
IIUC, for the kernel, either compiler should be fine. The
documentation[0] for compiling the RPi Zero kernel seems to bear this
out - it even uses the "CROSS_COMPILE=arm-linux-gnueabihf-". If you've
got access to the hardware it should be easy to test before kicking off
the Debian kernel package build.

Raspbian uses own definition of armhf (armv6) while Debian (and other
distros) use armv7 for armhf.

So using any documentation which mentions armhf and r/pi needs to be
done in careful way.

[Andrew M.A. Cater]

Post by Marcin Juszkiewicz

Post by Punit Agrawal
IIUC, for the kernel, either compiler should be fine. The
documentation[0] for compiling the RPi Zero kernel seems to bear this
out - it even uses the "CROSS_COMPILE=arm-linux-gnueabihf-". If you've
got access to the hardware it should be easy to test before kicking off
the Debian kernel package build.

Raspbian uses own definition of armhf (armv6) while Debian (and other
distros) use armv7 for armhf.
So using any documentation which mentions armhf and r/pi needs to be done in
careful way.

The (early) Raspberry Pi 0/1 (and the Pi Zero v1 at least) are armv6 with
hardware floating point when all the Linux distributions had already
decided on v7 and hardware floating point as armhf architecture.
Debian armel runs fine without recompilation at a slight penalty.

Raspbian history is effectively a port by interested Debian folk -
Raspberry Pi OS is subtly different (and includes non-free software
as part of the standard build).

Later models of Raspberry Pi run as v7 and hardware floating point so
Debian armhf runs natively.

Raspberry Pi OS is slowly moving to 64 bit but the earliest Pis are 32 bit
only. Debian arm64 will run natively on Raspberry Pi 64 bit - the latest
firmware vote will also mean that we can include Raspberry Pi firmware
in the installer for next version Bookworm.

Hope this helps,

Andy Cater

[Diederik de Haas]

Post by Andrew M.A. Cater
Debian armel runs fine without recompilation at a slight penalty.

It is exactly this 'slight penalty' that I want to verify.

I have a Zero W running Debian's armel kernel and I find that device to be
annoyingly slow.
I also have a RPi 1 running raspbian.org's 4.9 kernel, which is a special
kernel build by plugwash and compiled like the rest of raspbian.org packages,
which I do not find (annoyingly) slow.

[Arnd Bergmann]

Post by Diederik de Haas

Post by Andrew M.A. Cater
Debian armel runs fine without recompilation at a slight penalty.

It is exactly this 'slight penalty' that I want to verify.
I have a Zero W running Debian's armel kernel and I find that device to be
annoyingly slow.
I also have a RPi 1 running raspbian.org's 4.9 kernel, which is a special
kernel build by plugwash and compiled like the rest of raspbian.org packages,
which I do not find (annoyingly) slow.

The difference between the two kernels is almost certainly unrelated
to the toolchain they are built with, and entirely based on the
kernel patches that are added on top of mainline as well as the
configuration.

If there is a huge difference, my first guess would be the CPU frequency
scaling driver: if the CPU runs a different operating point, that
would have a much larger impact than anything the kernel itself
can do. Another obvious difference may be the GPU support: if you
the HDMI output, having an accelerated driver for it will feel
much faster than running on a plain framebuffer.

If you have a fast Raspbian kernel and a slow Debian kernel for
the same source version, can you take a look at the /boot/config-*
files, and the list of loaded kernel modules, to see if there
are any obvious differences?

Arnd

[Diederik de Haas]

Post by Arnd Bergmann

Post by Diederik de Haas
I have a Zero W running Debian's armel kernel and I find that device to be
annoyingly slow.
I also have a RPi 1 running raspbian.org's 4.9 kernel, which is a special
kernel build by plugwash and compiled like the rest of raspbian.org
packages, which I do not find (annoyingly) slow.

The difference between the two kernels is almost certainly unrelated
to the toolchain they are built with,

That surprises me as I thought that raspbian.org's effort (and reason for
existing at all) was entirely based on that.

Post by Arnd Bergmann
and entirely based on the kernel patches that are added on top of mainline
as well as the configuration.
If there is a huge difference, my first guess would be the CPU frequency
scaling driver: if the CPU runs a different operating point, that
would have a much larger impact than anything the kernel itself
can do.

In https://salsa.debian.org/kernel-team/linux/-/merge_requests/381 I did
enable the needed components for that. I will try whether the performance
governor makes a notable difference.
RPi 1 only has 1 frequency, but RPi 0 has 2.

Post by Arnd Bergmann
Another obvious difference may be the GPU support

They are both used headlessly, so that should be irrelevant.
The RPi 0W is attached to my smart energy meter and the RPi 1 I essentially
only use(d) for development purposes.

Post by Arnd Bergmann
If you have a fast Raspbian kernel and a slow Debian kernel for
the same source version, can you take a look at the /boot/config-*
files, and the list of loaded kernel modules, to see if there
are any obvious differences?

Can `=m` vs `=y` have a (relevant) impact?
I will ofc compare them, but I have already looked (a while ago) at the config
for obvious missing components and only found 1 which is now fixed:
https://salsa.debian.org/kernel-team/linux/-/merge_requests/486

Cheers,
Diederik

[Diederik de Haas]

Post by Arnd Bergmann
and entirely based on the kernel patches that are added on top of mainline
as well as the configuration.

The patch set may be more substantial then I initially thought.
In 2015 I asked plugwash the general procedure to make a raspbian.org style
kernel and my notes of that are attached.

Through that, I found that an 'update-rpi-patches' script was used (attached)
and that adds https://github.com/raspberrypi/linux.git as remote.

I don't think I ever succeeded in building such a kernel though, probably also
because I had ~0 experience with building a kernel at all.

[Arnd Bergmann]

Post by Punit Agrawal

Post by Diederik de Haas
But it can be that I'm looking at this problem all wrong and/or have some
tunnel vision towards building a cross compiler.
That is an important reason for making this ML thread.

IIUC, for the kernel, either compiler should be fine. The
documentation[0] for compiling the RPi Zero kernel seems to bear this
out - it even uses the "CROSS_COMPILE=arm-linux-gnueabihf-". If you've
got access to the hardware it should be easy to test before kicking off
the Debian kernel package build.

That is correct: it's the same compiler backend, the only difference
between an arm-linux-gnuabihf- and an arm-linux-gnueabi- toolchain are
which CPU and floating pointing ABI are set when building with the
default flags. The kernel itself is always built as soft-float even
for armv7 using an armhf toolchain, and the CPU is always set to
the minimum configured target machine, so both compilers will produce
the same kernel binary.

Note that at the moment, you can build a kernel that works on
both armv6 and armv7 CPUs, or armv7 and armv8, but not armv5
and armv6/v7, or armv6 and armv8, and this limits what machines
can be enabled in a single kernel. I have a prototype kernel
patch that intends to change the way we build armv6 kernels,
so one can actually enable armv6 non-SMP support (Raspberry
Pi 0/1, OMAP2, i.MX3, s3c64xx, ...) in the default armel
kernel along with the ARMv5 targets (orion, kirkwood, pxa,
at91, omap1, imx2, suniv, ...), but no longer with ARMv7.

Arnd

[Diederik de Haas]

Post by Arnd Bergmann

Post by Punit Agrawal
IIUC, for the kernel, either compiler should be fine. The
documentation[0] for compiling the RPi Zero kernel seems to bear this
out - it even uses the "CROSS_COMPILE=arm-linux-gnueabihf-". If you've
got access to the hardware it should be easy to test before kicking off
the Debian kernel package build.

How can I test that? I'm not a C programmer and I thought that crashes due to
arch differences would only manifest themselves under certain circumstances
which could take a while to manifest.

Post by Arnd Bergmann
That is correct: it's the same compiler backend, the only difference
between an arm-linux-gnuabihf- and an arm-linux-gnueabi- toolchain are
which CPU and floating pointing ABI are set when building with the
default flags. The kernel itself is always built as soft-float even
for armv7 using an armhf toolchain, and the CPU is always set to
the minimum configured target machine, so both compilers will produce
the same kernel binary.

So this means I *can* actually use the arm-linux-gnuabihf compiler* already
available in the Debian archives? I did not expect that as I _assumed_ it
would make binaries for Debian's armhf architecture.

*) I normally don't take things RPi Foundation puts out at face value