RFC: dropping armel from Debian for the upcoming release

[John Paul Adrian Glaubitz]

Hi Hector,

Upstream projects, ARM companies which I was able to check with, do
not care that much about maintaining old code for ARMv5t chipsets,
therefore supporting it is more and more costly resource wise (not
only in Debian).
Timely to the writing of this email, Arnd Bergmann posted the
following timeline to deprecate ARM (armel) architecture, you can read
Should Debian drop armel from the upcoming Debian release?

While I understand the reasoning behind it, also having read Arnd's mail, I don't
understand why these decisions rarely consider the environment and sustainability
when talking about removing support for a given hardware.

ARM5vT/armel devices are still widely used for various embedded and low-energy
applications where compute power doesn't matter but just reliability and energy
consumption.

Dropping support for these devices will mean that a lot of hardware will eventually
be thrown away which otherwise could still have been fulfilled their purpose without
any problems.

It's certainly not easy to determine the actual usage statistics, but as long as there
is a considerable user base, I think dropping support for hardware because it's old
doesn't sound right to me.

Adrian

[Arnd Bergmann]

can track follow up emails at debian-arm ML if interested. ]
Dear fellow developers,
- https://d-i.debian.org/daily-images/armel/
Debian Linux kernel packages are only building support for Raspberry
-
https://salsa.debian.org/kernel-team/linux/-/blob/master/debian/config/armel/defines.toml
while many other platforms have been dropped.
Upstream projects, ARM companies which I was able to check with, do
not care that much about maintaining old code for ARMv5t chipsets,
therefore supporting it is more and more costly resource wise (not
only in Debian).
Timely to the writing of this email, Arnd Bergmann posted the
following timeline to deprecate ARM (armel) architecture, you can read

To clarify the scope of my proposal, as that may be easy to
misunderstand: There is no current plan to drop kernel support
for any of the hundreds of ARMv5 machines that use devicetree,
or remove features that armel depends on.

The only ARMv5/v6 machines that are up for debate at this
point are the Nokia N800/N810 tablet because of its unusual
CPU (ARM1136r0), and a the few PXA2xx and Orion5x machine
that were never converted to DT. If there are actual users,
we'll try to keep them and change them to DT.

The timing I suggested for dropping the non-DT orion5x
machines in 6.13 is intended to still allow the debianonbuffalo
project to support all NAS boxes with a kernel using the
Trixie kernel source package with a custom config. Most
of the ARMv5 machines they support do use DT and will
keep working in future releases.

Should Debian drop armel from the upcoming Debian release?

This is of course a valid question to ask regardless.

It's clear that armel is used much less than armhf, but
I think it's also still more widely used than any of the
inofficial ports, so the question is when it hits that
threshold of not being worth maintainer time.

Most of the current users are probably fine with armel
being moved from a release architecture to ports, but from
a user perspective I also think it would be nice to do
that after Trixie is out, so that there is at least one
official release with time64 library support.

Arnd

[Martin]

Post by Arnd Bergmann
Most of the current users are probably fine with armel
being moved from a release architecture to ports, but from
a user perspective I also think it would be nice to do
that after Trixie is out, so that there is at least one
official release with time64 library support.

Good point!

Also, for armel to stay alive longer, we might like to remove some huge
applications from its small shoulders. (I believe, e.g. libreoffice is
still available for armel and I wonder, how useful that is.)

[Hector Oron]

Hello Adrian,

On Thu, 1 Aug 2024 at 09:39, John Paul Adrian Glaubitz

Post by John Paul Adrian Glaubitz

Should Debian drop armel from the upcoming Debian release?

While I understand the reasoning behind it, also having read Arnd's mail, I don't
understand why these decisions rarely consider the environment and sustainability
when talking about removing support for a given hardware.
ARM5vT/armel devices are still widely used for various embedded and low-energy
applications where compute power doesn't matter but just reliability and energy
consumption.
Dropping support for these devices will mean that a lot of hardware will eventually
be thrown away which otherwise could still have been fulfilled their purpose without
any problems.
It's certainly not easy to determine the actual usage statistics, but as long as there
is a considerable user base, I think dropping support for hardware because it's old
doesn't sound right to me.

The main reasoning for dropping the port are the problems listed at:
- https://release.debian.org/testing/arch_qualify.html

On the other hand, I find your arguments and Arnd's one about keeping
an official release with time64 library support very useful to keep
port alive one more release.

Note that dropping from stable does not mean we fully drop the port,
it can be kept as non-release architecture, then it would not block
security updates like the ones we had with linux kernel build
failures.

And for old devices (which I also still have) you can always find
unsupported releases at archive.debian.org, maybe we could have an
unofficial kernel package for those devices.

Regards,

[John Paul Adrian Glaubitz]

Hi Hector,

Post by Hector Oron

Post by John Paul Adrian Glaubitz
It's certainly not easy to determine the actual usage statistics, but as long as there
is a considerable user base, I think dropping support for hardware because it's old
doesn't sound right to me.

- https://release.debian.org/testing/arch_qualify.html

OK, so the main issue here seems to be the aging hardware of the buildds.

Post by Hector Oron
From what I remember from my discussions with Alex Graf (former colleague

at SUSE), there are some ARM64 systems which support 32-bit ARM binaries
without limitations.

Has that changed in the mean time?

Post by Hector Oron
On the other hand, I find your arguments and Arnd's one about keeping
an official release with time64 library support very useful to keep
port alive one more release.

Yeah, it would be strange if we dropped armel now after we've gone through
all the efforts to switch the port to time64_t ;-).

Post by Hector Oron
Note that dropping from stable does not mean we fully drop the port,
it can be kept as non-release architecture, then it would not block
security updates like the ones we had with linux kernel build
failures.

Sure, I'm aware of that. The problem is just that the infrastructure we have
at Debian Ports is quite inferior to what we have for release architectures
(no support for cruft, no Britney, no Ben etc), so moving an architecture
to Ports doesn't just mean you lose support for a stable release but also
quite a lot of stuff that is makes maintaining an architecture in Debian much
easier.

Post by Hector Oron
https://www.netbsd.org/ports/#tiers

But so far there has been little to no feedback to this idea. If we introduced tiers,
we could easily downgrade architectures like armel to a less well supported tier without
all the disadvantages that moving it to Debian Ports has.

Post by Hector Oron
And for old devices (which I also still have) you can always find
unsupported releases at archive.debian.org, maybe we could have an
unofficial kernel package for those devices.

Sure, that was never the question.

Adrian

[Arnd Bergmann]

Post by John Paul Adrian Glaubitz

Post by Hector Oron

Post by John Paul Adrian Glaubitz
It's certainly not easy to determine the actual usage statistics, but as long as there
is a considerable user base, I think dropping support for hardware because it's old
doesn't sound right to me.

- https://release.debian.org/testing/arch_qualify.html

OK, so the main issue here seems to be the aging hardware of the buildds.

Post by Hector Oron
From what I remember from my discussions with Alex Graf (former colleague

at SUSE), there are some ARM64 systems which support 32-bit ARM binaries
without limitations.
Has that changed in the mean time?

I believe the builds are all done on arm64 hardware, and the table
lists the same issue for available hardware on armel, armhf and arm64.

There are a few instructions that armel binaries can use that
are missing on armv8 hardware (cp15 barriers, swp/swpb atomics),
and a few differences in how the kernel treats unaligned memory
access and personality. Emulation for these is a bit tricky to
set up correctly for individual developers, but but the
build servers should generall just work.

Most of issues with armel packages are about libatomic link
time requirements, and about applications that hardcode armv7
or vfp instruction extensions.

Arnd

[John Paul Adrian Glaubitz]

Post by Arnd Bergmann
I believe the builds are all done on arm64 hardware, and the table
lists the same issue for available hardware on armel, armhf and arm64.
There are a few instructions that armel binaries can use that
are missing on armv8 hardware (cp15 barriers, swp/swpb atomics),
and a few differences in how the kernel treats unaligned memory
access and personality. Emulation for these is a bit tricky to
set up correctly for individual developers, but but the
build servers should generall just work.

Thanks for the clarification.

Post by Arnd Bergmann
Most of issues with armel packages are about libatomic link
time requirements, and about applications that hardcode armv7
or vfp instruction extensions.

Code that hardwires CPU baselines is considered RC-buggy anyway, so
that shouldn't be an argument. And that notoric libatomic issue is
just something that should be finally fixed in GCC upstream (unless
I am missing something why that can't be done).

Adrian

[Arnd Bergmann]

Post by John Paul Adrian Glaubitz

Post by Arnd Bergmann
Most of issues with armel packages are about libatomic link
time requirements, and about applications that hardcode armv7
or vfp instruction extensions.

Code that hardwires CPU baselines is considered RC-buggy anyway, so
that shouldn't be an argument. And that notoric libatomic issue is
just something that should be finally fixed in GCC upstream (unless
I am missing something why that can't be done).

Right, in principle none of this is specific to armel, but
I think this one gets noticed more than the others:

- upstream projects are likely to hardwire armv7 assembly for
arm32 than hitting the same problem on other architectures
since most people don't understand the exact cpu specific
requirements, and armv7 is what everyone has.

- for libatomic, I think it's the only remaining official
port that is affected (not sure about mips64el), while a
number of the inofficial ones likely have the same issue
(parisc, sparc, ...).

If none of the official release architectures had
the libatomic issue, it would make life a little easier
for their maintainers, but it makes maintaining the
non-release architectures harder because then fewer
people care.

I don't know if a gcc change can address all of the
libatomic issues, but that would of course be best.

Arnd

[Sebastian Ramacher]

Hi Héctor

can track follow up emails at debian-arm ML if interested. ]
Dear fellow developers,
- https://d-i.debian.org/daily-images/armel/
Debian Linux kernel packages are only building support for Raspberry
- https://salsa.debian.org/kernel-team/linux/-/blob/master/debian/config/armel/defines.toml
while many other platforms have been dropped.
Upstream projects, ARM companies which I was able to check with, do
not care that much about maintaining old code for ARMv5t chipsets,
therefore supporting it is more and more costly resource wise (not
only in Debian).
Timely to the writing of this email, Arnd Bergmann posted the
following timeline to deprecate ARM (armel) architecture, you can read
Should Debian drop armel from the upcoming Debian release?

Was there a conclusion to the discussion on d-***@l.d.o? What is the
opinion of the two porters that we currently have listed for armel
(added to CC)?

Cheers

[Chris Hofstaedtler]

Hi,

Post by Sebastian Ramacher

Should Debian drop armel from the upcoming Debian release?

opinion of the two porters that we currently have listed for armel
(added to CC)?

fakeroot, a notable key part of the infrastructure required to build
packages, FTBFS on armel and armhf since March, due to t64 changes
on these architectures.

I think this is indicative of the developer interest in armel and
armhf and the resulting available resources in Debian at this time.

Chris

[Martin]

Post by Chris Hofstaedtler
fakeroot, a notable key part of the infrastructure required to build
packages, FTBFS on armel and armhf since March, due to t64 changes
on these architectures.
I think this is indicative of the developer interest in armel and
armhf and the resulting available resources in Debian at this time.

My employer is a user of armel and armhf, but we were not aware of the
problem. I'm pretty sure, my boss had offered a "bug bounty" or
something, if I had insisted.

What would be the best way, esp. for people outside of Debian, to always
know about such problems? And not only read about it, when it's already
solved?

Cheers

[Paul Gevers]

Hi,

Post by Martin
What would be the best way, esp. for people outside of Debian, to always
know about such problems? And not only read about it, when it's already
solved?

Ideally bug affecting specific architectures should have the right
usertags. I believe in this case the bug had it:

https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=debian-***@lists.debian.org

Paul

[Adrian Bunk]

Post by Chris Hofstaedtler
Hi,

Post by Sebastian Ramacher

Should Debian drop armel from the upcoming Debian release?

opinion of the two porters that we currently have listed for armel
(added to CC)?

fakeroot, a notable key part of the infrastructure required to build
packages, FTBFS on armel and armhf since March, due to t64 changes
on these architectures.
I think this is indicative of the developer interest in armel and
armhf and the resulting available resources in Debian at this time.

I think this is more indicative of fakeroot being in the not uncommon
situation that this is a key part of the Debian infrastructure only
one person in Debian knows well.

I am surprised that there were uploads by the maintainer (who is also
upstream) that ignored this bug.

I am also surprised that the people from Canonical who were doing the
time_t transition ignored this bug.

It is laudable that you finally fixed it, but I'd guess you'd agree that
this required fakeroot knowledge and not arm knowledge.

And you cannot single out armel and armhf for this in general,
if the level of porter activity you expect would be a prerequisite
in trixie then not a single of the bookworm architectures would
qualify.

We do have a porter problem, but hppa, riscv64 and loong64 are the only
potential architectures for trixie if continuous and proactive activity
by several porters is a hard requirement.

Post by Chris Hofstaedtler
Chris

cu
Adrian

BTW:
It looks bogus that amd64 is exempted from the porter requirement based on
Our toolchain maintainers are happy to support amd64 as-is.
Who is responsible for fixing amd64 specific issues in non-toolchain key
parts of the infrastructure?

[Hector Oron]

Hello Sebastian,

(Apologies for delay, I starting drafting the email, but I had to
leave and never got back to it until now)
There was a compelling reason to do at least one more armel release to
have at least one official release with time64 support.
There were other comments to keep architecture for environmental
reasons, to keep supporting old devices that are still in the field,
there are lots of {sheeva,dream,*}plugs based around Marvell's
kirkwood and many NAS devices based on Orion chipsets. Martin Michmayr
requested a readdition of OpenRD in stable debian-installer
(https://bugs.debian.org/1068898), maybe nice to keep for this one
last release.
There was also a suggestion to mark non-for-us or stop building large
packages that may not be that useful in the armel architecture, such
LibreOffice, possibly some desktops, etc.. since armel is mostly meant
for embedded and headless devices (may be some with small
screen/display).

Post by Sebastian Ramacher
What is the
opinion of the two porters that we currently have listed for armel
(added to CC)?

The two porters listed in the arch qualification page missed the thread.
However, since packages are building on arm64 hardware, until the
hardware is capable of doing so, we should at least try to keep armel
for the Trixie release.

Regards
--
Héctor Orón -.. . -... .. .- -. -.. . ...- . .-.. --- .--. . .-.

[Jeffrey Walton]

Post by Hector Oron
[...]
There was also a suggestion to mark non-for-us or stop building large
packages that may not be that useful in the armel architecture, such
LibreOffice, possibly some desktops, etc.. since armel is mostly meant
for embedded and headless devices (may be some with small
screen/display).

I think this is a good idea.

I wonder if it can be generalized and applied to other architectures,
like PowerPC. (I am also aware of PowerPC being used in high-end
workstations, like Talos
<https://www.raptorcs.com/content/base/products.html>).

Jeff

[Emanuele Rocca]

Hello Hector,
thanks for bringing this up!

Post by Hector Oron
There was a compelling reason to do at least one more armel release to
have at least one official release with time64 support.

FWIW I think this makes a lot of sense: have one stable release with
time64 support, move to ports after Trixie.

[Martin]

Post by Emanuele Rocca
Hello Hector,
thanks for bringing this up!

¡Gracias, Héctor!

Post by Emanuele Rocca

Post by Hector Oron
There was a compelling reason to do at least one more armel release to
have at least one official release with time64 support.

FWIW I think this makes a lot of sense: have one stable release with
time64 support, move to ports after Trixie.

Yes.

[Rick Thomas]

Plus 1 from me!
Rick

Post by Emanuele Rocca
Hello Hector,
thanks for bringing this up!

Post by Hector Oron
There was a compelling reason to do at least one more armel release to
have at least one official release with time64 support.

FWIW I think this makes a lot of sense: have one stable release with
time64 support, move to ports after Trixie.

[Adrian Bunk]

Post by Hector Oron
...
There was also a suggestion to mark non-for-us or stop building large
packages that may not be that useful in the armel architecture, such
LibreOffice, possibly some desktops, etc.. since armel is mostly meant
for embedded and headless devices (may be some with small
screen/display).
...

This doesn't strike me as coming from people who have an overview of
actual issues - it would not solve any problem but would add new
problems.

armel does not have any buildd speed issues.

It is REALLY preferable to attempt to build as much as possible even on
the most obscure ports architectures, since working around something not
being available is manual work.

But e.g. building libflac needs nearly the complete Haskell ecosystem.

The build dependencies of something like git require subversion and Qt
and Java and tons of other stuff.

You can workaround such issues, but that is manual work and we are not
gaining anything.

Post by Hector Oron
Regards

cu
Adrian

[Adrian Bunk]

...
What is the opinion of the two porters that we currently have listed
for armel
...

I am not in favour of removing armel in trixie.

Cheers

cu
Adrian