Releasing linux/6.1.52-1 bookworm-security update without armel build, Image size problems

[Salvatore Bonaccorso]

Hi all,

We have problem with the image size of armel builds in bookworm. There
is a pending bookworm-security linux update pending which is currently
blocked due to armel FTBFS due to the image size increase:

https://people.debian.org/~carnil/buildd-logs/linux/linux_6.1.52-1_armel-2023-09-07T08:53:41Z.gz

debian/bin/buildcheck.py debian/build/build_armel_none_marvell armel none marvell
Can't read ABI reference. ABI not checked!
Image size 2753652/2729712, using 100.88%. Too large. Refusing to continue.
make[2]: *** [debian/rules.real:169: debian/stamps/build_armel_none_marvell] Error 1
make[2]: Leaving directory '/<<PKGBUILDDIR>>'
make[1]: *** [debian/rules.gen:1615: build-arch_armel_none_marvell_real_image] Error 2
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
make: *** [debian/rules:39: build-arch] Error 2
dpkg-buildpackage: error: debian/rules binary-arch subprocess returned exit status 2

In fact we are already too narrow to 100% in any case, but there was a
bump between 6.1.41 and 6.1.42 upstream AFAICS:

6.1.52-1 Image size 2751596/2729712, using 100.80%. Too large. Refusing to continue.
6.1.51-1 Image size 2752212/2729712, using 100.82%. Too large. Refusing to continue.
6.1.47-1 Image size 2752676/2729712, using 100.84%. Too large. Refusing to continue.
6.1.45-1 Image size 2751292/2729712, using 100.79%. Too large. Refusing to continue.
6.1.43-1 Image size 2751348/2729712, using 100.79%. Too large. Refusing to continue.
6.1.42-1 Image size 2752924/2729712, using 100.85%. Too large. Refusing to continue.
6.1.41-1 Image size 2701348/2729712, using 98.96%. Image fits. Continuing.
6.1.40-1 Image size 2703956/2729712, using 99.06%. Under 1% space in UNRELEASED. Continuing.
6.1.38-1 Image size 2703076/2729712, using 99.02%. Under 1% space in bookworm. Continuing.

I doupt anybody is sensibly using armel nowdays under bookworm, so my proposed
course of action for unblock the bookworm-security update is:

Either

- ignore the image size and implicitly drop support for devices which would break
due to size constraints, the current upper limit is adjusted for the following:

# Buffalo Linkstation LS-WSXL/WXL/WVL (from stock kernel): 2729776 - 64 = 2729712

or:

- Relese the DSA without armel builds. This is not optimal and for the point release
we need to have to have all builds, but this gives people who still are interested
in this architecture to step up and propose a fix for the problem, otherwise then
disable the image size check, and then effectively dropping some support.

Attached is the result of bloat-o-meter script between 6.1.41 and 6.1.42.

I might put me in a bad spot, but should have been support for armel been
dropped earlier and should we do it for trixie following the same done for
mipsel?

Note that the last time the problem arised already earlier in
experimental and Ben workarounded it there with
https://salsa.debian.org/kernel-team/linux/-/commit/9dfe6d33a4fd220394228b30cbbfdb3b444d36ec
We probably can do that as well here. 60443c88f3a8 ("kallsyms: Improve
the performance of kallsyms_lookup_name()") was in fact backported to
6.1.42. So this is next I would try and disable MPTCP and
FUNCTION_TRACER. But the problem with armel will remain.

armel people, can you have ideally look at it ASAP on the comments
please, I would not like to delay the DSA for linux on
bookworm-security too much.

Thanks for having a look,

Regards,
Salvatore

[Adrian Bunk]

Post by Salvatore Bonaccorso
...
- Relese the DSA without armel builds. This is not optimal and for the point release
we need to have to have all builds, but this gives people who still are interested
in this architecture to step up and propose a fix for the problem, otherwise then
disable the image size check, and then effectively dropping some support.
...
armel people, can you have ideally look at it ASAP on the comments
please, I would not like to delay the DSA for linux on
bookworm-security too much.

Releasing this DSA without armel and sorting out the issue for the point
release sounds like the best option to me.

Post by Salvatore Bonaccorso
Thanks for having a look,
Regards,
Salvatore

cu
Adrian

[Salvatore Bonaccorso]

Hi,

Post by Adrian Bunk

Post by Salvatore Bonaccorso
...
- Relese the DSA without armel builds. This is not optimal and for the point release
we need to have to have all builds, but this gives people who still are interested
in this architecture to step up and propose a fix for the problem, otherwise then
disable the image size check, and then effectively dropping some support.
...
armel people, can you have ideally look at it ASAP on the comments
please, I would not like to delay the DSA for linux on
bookworm-security too much.

Releasing this DSA without armel and sorting out the issue for the point
release sounds like the best option to me.

FWIW, following Ben's aproach for unstable, here is my proposed change
for bookworm in the near-term:

https://salsa.debian.org/kernel-team/linux/-/merge_requests/844

I have verified by cross-building that the image size goes down to

Image size 2644124/2729712, using 96.86%. Image fits. Continuing.

which would be sufficient so far.

So we can at least include the above for the point release and
releasing the DSA earlier without the armel builds.

Thank you!

Regards,
Salvatore

[Domenico Andreoli]

Post by Salvatore Bonaccorso
Hi,

Post by Adrian Bunk

Post by Salvatore Bonaccorso
...
- Relese the DSA without armel builds. This is not optimal and for the point release
we need to have to have all builds, but this gives people who still are interested
in this architecture to step up and propose a fix for the problem, otherwise then
disable the image size check, and then effectively dropping some support.
...
armel people, can you have ideally look at it ASAP on the comments
please, I would not like to delay the DSA for linux on
bookworm-security too much.

Releasing this DSA without armel and sorting out the issue for the point
release sounds like the best option to me.

FWIW, following Ben's aproach for unstable, here is my proposed change
https://salsa.debian.org/kernel-team/linux/-/merge_requests/844
I have verified by cross-building that the image size goes down to
Image size 2644124/2729712, using 96.86%. Image fits. Continuing.
which would be sufficient so far.
So we can at least include the above for the point release and
releasing the DSA earlier without the armel builds.

Looks good to me. Thanks!

Post by Salvatore Bonaccorso
Thank you!
Regards,
Salvatore

[Paul Gevers]

Hi Salvatore,

Post by Salvatore Bonaccorso
but should have been support for armel been
dropped earlier and should we do it for trixie

The kernel for armel went over some hardware limits before (I was
affected with my NAS, where I couldn't upgrade the kernel to bullseye as
documented in the release notes [1]). Is the current situation reaching
the limit for all armel devices, or "just" for some and are the others
probably fine for some years to come?

If we're now reaching the final limit and if it was foreseeable that we
would reach that limit, then yes it would have made sense to drop armel
*before* the bookworm release, but alas. If the kernel team can't
support the kernel on armel, than armel shouldn't be a release
architecture for trixie. If it's only some devices, than we "just" need
to communicate that clearly.

I don't have a clear advice for the current situation in security and
the next point release, let's hope you can stretch the situation a bit
longer. I recall that the kernel package has safety checks in place and
refuses to *try* to install the kernel if it doesn't fit on the
hardware. That means that you don't cripple the hardware of affected
people, but "merely" can't give them security support? I guess it would
be possible (as long as support lasts; no LTS support) for effected
systems to run the security supported bullseye kernel.

Paul

[1]
https://www.debian.org/releases/bullseye/armel/release-notes/ch-information.en.html#no-longer-supported-hardware

[Bastian Blank]

Hi

Post by Paul Gevers
If we're now reaching the final limit and if it was foreseeable that we
would reach that limit, then yes it would have made sense to drop armel
*before* the bookworm release, but alas. If the kernel team can't support
the kernel on armel, than armel shouldn't be a release architecture for
trixie. If it's only some devices, than we "just" need to communicate that
clearly.

We have two armel kernel currently:
- "marvell", for some CPU from Marvell, and
- "rpi", for Raspberry Pi 1 and related devices.

The first one is the one with included size limitations, because those
load the kernel from a pre-defined flash partition, whose size can't be
easily changed by the user. This one is now overflowing for the second
to last documented one in the kernel package config.

The second one is for the original Raspberry Pi 1 type. There we don't
have any size limits, as the kernel is loaded from a file system.
However those systems contain a ARMv6 CPU. So our armel port is only
partially usable anyway, as is is built for ARMv4. There exists with
Raspbian a better suited forked distribution with ARMv6 as target.

So yes there is a small number of devices we can still support with the
armel port, but where we are a bad choice.

Everything newer is ARMv7, supported by the armhf port, or ARMv8,
supported by the arm64 port.

Latest popcon for stable is:

linux-image-marvell: 31
linux-image-rpi: 7

Debian itself does not have any armel hardware. Everything is done on
armhf or arm64. Sadly the armhf supporting systems are already in the
progress of drying up. Even some ARMv8 vendors do not longer include
32bit support.

Bastian

[Andrew M.A. Cater]

Post by Bastian Blank
Hi

Post by Paul Gevers
If we're now reaching the final limit and if it was foreseeable that we
would reach that limit, then yes it would have made sense to drop armel
*before* the bookworm release, but alas. If the kernel team can't support
the kernel on armel, than armel shouldn't be a release architecture for
trixie. If it's only some devices, than we "just" need to communicate that
clearly.

- "rpi", for Raspberry Pi 1 and related devices.
The second one is for the original Raspberry Pi 1 type. There we don't
have any size limits, as the kernel is loaded from a file system.
However those systems contain a ARMv6 CPU. So our armel port is only
partially usable anyway, as is is built for ARMv4. There exists with
Raspbian a better suited forked distribution with ARMv6 as target.

No - Raspbian contains commercial software now by default. We have to use
Raspberry Pi firmware but Raspberry pi is *not* purely ARM v6 it's ARM v6 with
hardware floating point - and therefore incompatible with Debian and every
other ARM v6 version. Although it is less than optimal, it is still perfectly
supportable by all the packages in armel.

Post by Bastian Blank
So yes there is a small number of devices we can still support with the
armel port, but where we are a bad choice.

See above: note that the Raspberry Pi Zero (original version) is still very
much in production and is 32 bit and ARM v6 hardware floating point. It is
still a prime target for armel - and the number of devices produced is not
small.

Post by Bastian Blank
Everything newer is ARMv7, supported by the armhf port, or ARMv8,
supported by the arm64 port.
linux-image-marvell: 31
linux-image-rpi: 7
Debian itself does not have any armel hardware. Everything is done on
armhf or arm64. Sadly the armhf supporting systems are already in the
progress of drying up. Even some ARMv8 vendors do not longer include
32bit support.

This is, unfortunately, the case. I'm not sure where our sd card images
are built but the Raspberry Pi iamges are built from Gunnar Wolf's scripts
primarily for everything prior to RPi4.

All best, as ever,

Andy

Post by Bastian Blank
Bastian
--
Each kiss is as the first.
-- Miramanee, Kirk's wife, "The Paradise Syndrome",
stardate 4842.6

[Gianluca Renzi]

Post by Bastian Blank
The first one is the one with included size limitations, because those
load the kernel from a pre-defined flash partition, whose size can't be
easily changed by the user.  This one is now overflowing for the second
to last documented one in the kernel package config.

Seems like this would be solvable by writing a bootloader to the flash
partition that would be able to load Linux from a normal filesystem?

I think any modern (at least more than 10 years of lifespan) bootloader
(u-boot, redboot, barebox, etc.,...) can read from any supported
filesystem (at least ext2, ext3 and ext4 and maybe nand-related
filesystems like yaffs, yaffs2, ubifs), so having the {z|u}Image kernel
stored on those filesystems should be not an issue...

The biggest problem will be the update procedures for those hardware.
All scripts must be changed to reflect the new partition scheme. This is
*not* an easy fix.

Just my $0.02

Regards

Gianluca

[bodhi]

Post by Gianluca Renzi

Post by Bastian Blank
The first one is the one with included size limitations, because those
load the kernel from a pre-defined flash partition, whose size can't be
easily changed by the user. This one is now overflowing for the second
to last documented one in the kernel package config.

Seems like this would be solvable by writing a bootloader to the flash
partition that would be able to load Linux from a normal filesystem?

I think any modern (at least more than 10 years of lifespan) bootloader
(u-boot, redboot, barebox, etc.,...) can read from any supported
filesystem (at least ext2, ext3 and ext4 and maybe nand-related
filesystems like yaffs, yaffs2, ubifs), so having the {z|u}Image kernel
stored on those filesystems should be not an issue...
The biggest problem will be the update procedures for those hardware.
All scripts must be changed to reflect the new partition scheme. This is
*not* an easy fix.

I agreed with Paul and also Gianluca. I'm the u-boot maintainer for many Marvell Kirkwood SoC boards. All these boards are actively maintained and up to modern standard, i.e. can boot with file systems on SATA, USB, MMC... We can ignore what's currently in flash partitions and boot with kernels on disk rootfs.

What can I do to help in upgrading the update procedure to boot Debian 12 on Marvell Kirkwood SoCs? can somebody give me some pointer where to start looking for these scripts?

All the best,
Tony

Post by Gianluca Renzi
Just my $0.02
Regards
Gianluca
--
Eurek s.r.l. |
Electronic Engineering | http://www.eurek.it
via Celletta 8/B, 40026 Imola, Italy | Phone: [phone]
p.iva 00690621206 - c.f. 04020030377 | Fax: [phone]

[Philippe Clérié]

For what it's worth, if it helps I can pitch in and test.

I have a QNAP HS-210 and a TS-212P that I've retired since the first
time I read that Debian would drop armel arch. I know there have been
workarounds the limitations, and I keep having plans to implement at
least one. But! Good intentions and all! :-)

...
Philippe

------
The trouble with common sense is that it is so uncommon.
(Anonymous)

Post by bodhi

Post by Gianluca Renzi

Post by Bastian Blank
The first one is the one with included size limitations, because those
load the kernel from a pre-defined flash partition, whose size can't be
easily changed by the user. This one is now overflowing for the second
to last documented one in the kernel package config.

Seems like this would be solvable by writing a bootloader to the flash
partition that would be able to load Linux from a normal filesystem?

I think any modern (at least more than 10 years of lifespan) bootloader
(u-boot, redboot, barebox, etc.,...) can read from any supported
filesystem (at least ext2, ext3 and ext4 and maybe nand-related
filesystems like yaffs, yaffs2, ubifs), so having the {z|u}Image kernel
stored on those filesystems should be not an issue...
The biggest problem will be the update procedures for those hardware.
All scripts must be changed to reflect the new partition scheme. This is
*not* an easy fix.

I agreed with Paul and also Gianluca. I'm the u-boot maintainer for many Marvell Kirkwood SoC boards. All these boards are actively maintained and up to modern standard, i.e. can boot with file systems on SATA, USB, MMC... We can ignore what's currently in flash partitions and boot with kernels on disk rootfs.
What can I do to help in upgrading the update procedure to boot Debian 12 on Marvell Kirkwood SoCs? can somebody give me some pointer where to start looking for these scripts?
All the best,
Tony

Post by Gianluca Renzi
Just my $0.02
Regards
Gianluca
--
Eurek s.r.l. |
Electronic Engineering | http://www.eurek.it
via Celletta 8/B, 40026 Imola, Italy | Phone: [phone]
p.iva 00690621206 - c.f. 04020030377 | Fax: [phone]

[Domenico Andreoli]

Post by Bastian Blank
The first one is the one with included size limitations, because those
load the kernel from a pre-defined flash partition, whose size can't be
easily changed by the user.  This one is now overflowing for the second
to last documented one in the kernel package config.

Seems like this would be solvable by writing a bootloader to the flash
partition that would be able to load Linux from a normal filesystem?

Yes, if the support for the given device is mainlined.

I doubt this is the case for my QNAP though, would not dare to try.

[Ben Hutchings]

Post by Paul Gevers
Hi Salvatore,

Post by Salvatore Bonaccorso
but should have been support for armel been
dropped earlier and should we do it for trixie

The kernel for armel went over some hardware limits before (I was
affected with my NAS, where I couldn't upgrade the kernel to bullseye as
documented in the release notes [1]). Is the current situation reaching
the limit for all armel devices, or "just" for some and are the others
probably fine for some years to come?

The issue exists with many devices supported by the "marvell" flavour.
We also have an "rpi" flavour for armel that supports Raspberry Pi
models with Arm v6 CPUs, and that doesn't have any size limits that
we've needed to worry about yet.

Post by Paul Gevers
If we're now reaching the final limit and if it was foreseeable that we
would reach that limit, then yes it would have made sense to drop armel
*before* the bookworm release, but alas. If the kernel team can't
support the kernel on armel, than armel shouldn't be a release
architecture for trixie. If it's only some devices, than we "just" need
to communicate that clearly.

[...]

I would be quite happy to drop the "marvell" flavour for trixie. The
size limits are a recurring problem, and the supported SoCs and devices
seem to have been out of production for a long time. In contrast, the
Raspberry Pi models with v6 CPUs are still in production.

(I do have my doubts as to whether it will be possible to continue
supporting a useful user-space for armel, but I certainly can't speak
authoritatively about that.)

Ben.

[Domenico Andreoli]

Post by Ben Hutchings

Post by Paul Gevers
Hi Salvatore,

Post by Salvatore Bonaccorso
but should have been support for armel been
dropped earlier and should we do it for trixie

The kernel for armel went over some hardware limits before (I was
affected with my NAS, where I couldn't upgrade the kernel to bullseye as
documented in the release notes [1]). Is the current situation reaching
the limit for all armel devices, or "just" for some and are the others
probably fine for some years to come?

The issue exists with many devices supported by the "marvell" flavour.
We also have an "rpi" flavour for armel that supports Raspberry Pi
models with Arm v6 CPUs, and that doesn't have any size limits that
we've needed to worry about yet.

I happily delayed the issue by changing the layout of the mtd partitions
of my QNAP with a nice script [0]. It's able to handle a good number
of models and I think it's a great solution for the short/medium term.

Post by Ben Hutchings

Post by Paul Gevers
If we're now reaching the final limit and if it was foreseeable that we
would reach that limit, then yes it would have made sense to drop armel
*before* the bookworm release, but alas. If the kernel team can't
support the kernel on armel, than armel shouldn't be a release
architecture for trixie. If it's only some devices, than we "just" need
to communicate that clearly.

[...]
I would be quite happy to drop the "marvell" flavour for trixie. The
size limits are a recurring problem, and the supported SoCs and devices
seem to have been out of production for a long time. In contrast, the
Raspberry Pi models with v6 CPUs are still in production.

I would find reasonable if trixie dropped "marvell" flavor. The bell
for the armel QNAP devices has been warning for years already, anybody
caring of their data should move it somewhere else.

Post by Ben Hutchings
(I do have my doubts as to whether it will be possible to continue
supporting a useful user-space for armel, but I certainly can't speak
authoritatively about that.)
Ben.
--
Ben Hutchings - Debian developer, member of kernel, installer and LTS
teams

[0] https://github.com/amouiche/qnap_mtd_resize_for_bullseye

[Adrian Bunk]

Post by Salvatore Bonaccorso
...
Note that the last time the problem arised already earlier in
experimental and Ben workarounded it there with
https://salsa.debian.org/kernel-team/linux/-/commit/9dfe6d33a4fd220394228b30cbbfdb3b444d36ec
We probably can do that as well here. 60443c88f3a8 ("kallsyms: Improve
the performance of kallsyms_lookup_name()") was in fact backported to
6.1.42. So this is next I would try and disable MPTCP and
FUNCTION_TRACER.
...

Yes, that looks reasonable.

Additionally, one generic cause of bloat is:
debian/changelog: * Enable UNICODE. (closes: #985689)
debian/config/config:CONFIG_UNICODE=y

That's 74 kB uncompressed, and there doesn't seem to be any
justification for not making it modular. It's not urgent since
Bens change handles the immediate problem, but worth changing
in unstable.

cu
Adrian

[Salvatore Bonaccorso]

Hi Adrian,

Sorry for not replying early, busy with preparing the updates.

Post by Adrian Bunk

Post by Salvatore Bonaccorso
...
Note that the last time the problem arised already earlier in
experimental and Ben workarounded it there with
https://salsa.debian.org/kernel-team/linux/-/commit/9dfe6d33a4fd220394228b30cbbfdb3b444d36ec
We probably can do that as well here. 60443c88f3a8 ("kallsyms: Improve
the performance of kallsyms_lookup_name()") was in fact backported to
6.1.42. So this is next I would try and disable MPTCP and
FUNCTION_TRACER.
...

Yes, that looks reasonable.

Great thanks, this landed now for the point release udpates and in
fact we have the armel builds back.

Post by Adrian Bunk
debian/changelog: * Enable UNICODE. (closes: #985689)
debian/config/config:CONFIG_UNICODE=y
That's 74 kB uncompressed, and there doesn't seem to be any
justification for not making it modular. It's not urgent since
Bens change handles the immediate problem, but worth changing
in unstable.

Could you fill a bug against src:linux for it, so this might be
further addressed in unstable for armel?

Regards,
Salvatore

[Fan Naibed]

Post by Salvatore Bonaccorso
Hi all,

...

Post by Salvatore Bonaccorso
I doupt anybody is sensibly using armel nowdays under bookworm, so my proposed

...

FYI, An old, headless, raspberry pi (armel) with GPS and camera, on
Debian testing, was being sensibly used for long-term, slow, time-lapse
photography, where it was good enough for the intended purpose. Another
pi was slated for print server duties, but was mostly just idling
because the particular printer was not supported. Both stopped
responding over the network after upgrades some months ago, so this
decision wasn't involved; the problem was probably caused by systemd
network interface renaming. The hope was to get them going again with a
little debugging, or a re-install if necessary, as builds are still
being made[1]. It's always sad to see support have to be dropped for old
functioning hardware still sensibly serving purposes they can
accomplish. But, understandable with the size growth required to support
newer "better" hardware. Thanks for keeping them alive, and up to date,
as long as they were.


[1] https://raspi.debian.net/daily-images/