[Y2038] debian/armhf time64 build?

Post by Dominique MARTINET
Arnd did a superb recap in 2020, let me just link to it first in case anyone
https://lists.debian.org/debian-arm/2020/02/msg00024.html

(It was Steve who did the recap of the work that I had done)

Post by Dominique MARTINET
tl;dr: glibc 2.34 added a -D_TIME_BITS=64 toggle, but afaiu we need to
rebuild (virtually) everything to use it as that breaks the ABI for
anything using time_t
As far as I'm aware, there haven't been any new public discussion since
that 2020 thread -- has there been any new attempt I missed?

I am not aware of anyone trying to do another bootstrap after me.

Post by Dominique MARTINET
From what I can tell, the main thing that has changed is that there

is even less developer time that we can spend on armhf. With every
year that passes, more workloads are moving from 32-bit userspace
native 64 bit, so the number of people that is willing to help
out is going down, though that does not necessarily make it less
important to those that do care about 32 bit.

Another thing that has changed is that /other/ 32-bit distros are
getting dropped, notably Fedora 37 dropped 32-bit Armv7 support,
and Arch Linux dropped support for any platform without NEON
support. OpenWRT on the other hand just had their first release
that is using 64-bit time_t on all musl-based targets, though
not yet on i386+glibc.

Post by Dominique MARTINET
My naive opinion is that having a flag day and dealing with fallouts
might ultimately be the fastest way forward, but as there won't be any
safety check to detect incompatibility I don't think anyone will be
looking forward to the weird errors that will ensure for e.g. locally
compiled programs and whatsnot.
So that might be the first thing to think about again?
(... and I wish I had any idea there, but nothing practical to
suggest...)

I would agree that the idea of doing a soft migration that
correctly tracks library dependencies and allows a safe dist-upgrade
across the rebuild is pretty much dead because of the burden this
would put on package maintainers, so it will have to be fresh
time64 bootstrap.

Last time, the discussion got stuck because that means introducing
a new target name for dpkg, which then requires a new target triplet
to allow a multiarch installation. This in turn requires additional
work for porting packages that need to know about the new target
triplet. I don't think we can solve this problem now, but should
postpone the inevitable debate about this until someone has done
the work of rebootstrapping a working armhf environment with the
current names.

I don't expect that I'll have time to do this myself, but I'm
happy to help someone else figure out how to get to that point.

Arnd

Dominique MARTINET

2022-09-28 07:30:01 UTC

Post by Arnd Bergmann
(It was Steve who did the recap of the work that I had done)

(woops, sorry. Thank you both!)

I am not aware of anyone trying to do another bootstrap after me.
From what I can tell, the main thing that has changed is that there
is even less developer time that we can spend on armhf. With every
year that passes, more workloads are moving from 32-bit userspace
native 64 bit, so the number of people that is willing to help
out is going down, though that does not necessarily make it less
important to those that do care about 32 bit.
Another thing that has changed is that /other/ 32-bit distros are
getting dropped, notably Fedora 37 dropped 32-bit Armv7 support,
and Arch Linux dropped support for any platform without NEON
support. OpenWRT on the other hand just had their first release
that is using 64-bit time_t on all musl-based targets, though
not yet on i386+glibc.

Yes, we also released our first aarch64 board earlier this year (end of
2021 actually), but we still plan on using armv7 for a while longer so
we cannot just ignore that... Unfortunately as a very small company I'm
afraid the amount of help I can give is rather limited at this point.

Ugh, this is going to be a massive headache...
Other distributions I've worked with (e.g. nixos) have a wrapper for gcc
and clang that just enforce the flags they want the distro to be built
with -- I don't think debian has anything like that, would that be
easier to work with? My line of thinking is that there will only be a
single place to fix instead of configure/cmake/meson and all hand-made
build scripts that exist around here.

Alternatively this would be delaying things even further to get a new
glibc version, but have a glibc build option that just makes this the
default and rebuild that debian repo with it?
afaik that's what musl has done in commit 38143339646a ("switch all
existing 32-bit archs to 64-bit time_t"):
https://git.musl-libc.org/cgit/musl/commit/?id=38143339646a4ccce8afe298c34467767c899f51

Ultimately (after year 2038), building without the option would be
meaningless and there really should be a way to enable it by
default. There will ALWAYS be build systems that ignore your new target
and we would basically be breaking all of these....

I really think this should/could probably be discussed first; there
shouldn't be THAT many packages breaking once gcc/glibc/etc compile,
but if we cannot reach an agreement on how to enforce this option there
really is no point.
(I didn't add libc-alpha to this mail, but happy to start over with them
in the loop)

Post by Arnd Bergmann
I don't think we can solve this problem now, but should
postpone the inevitable debate about this until someone has done
the work of rebootstrapping a working armhf environment with the
current names.
I don't expect that I'll have time to do this myself, but I'm
happy to help someone else figure out how to get to that point.

I totally understand, thanks for all the work and the quick reply.

(I'll report back if I can allocate some time to help consistently, but
don't hold your breath on that)

--
Dominique

Arnd Bergmann

2022-09-28 10:10:01 UTC

Post by Dominique MARTINET
Ugh, this is going to be a massive headache...
Other distributions I've worked with (e.g. nixos) have a wrapper for gcc
and clang that just enforce the flags they want the distro to be built
with -- I don't think debian has anything like that, would that be
easier to work with? My line of thinking is that there will only be a
single place to fix instead of configure/cmake/meson and all hand-made
build scripts that exist around here.

Passing the flags to the C compiler is not the issue here, this can
be done in the gcc and clang packages directly by defining the right
set of macros (with the minor annoyance that glibc itself expects to
be built without those flags, but that again can obviously be solved).

Post by Dominique MARTINET
Alternatively this would be delaying things even further to get a new
glibc version, but have a glibc build option that just makes this the
default and rebuild that debian repo with it?
afaik that's what musl has done in commit 38143339646a ("switch all
https://git.musl-libc.org/cgit/musl/commit/?id=38143339646a4ccce8afe298c34467767c899f51

That is what I had proposed years ago when the glibc developers
started their work, but this was rejected as they do not wan to
change the default behavior for a given architecture, or make
the ABI conditional based on a compile-time flag during
compilation of the libc.

Again, this is just an implementation detail for getting things
running, the hard question that I want to postpone is what to do
about having both types of armhf debian packages (time32 and
time64) coexist on a single file system and instead focus on
the actual bootstrapping to find all the packages that break
just because of the time_t change.

Arnd

Jeffrey Walton

2022-09-28 13:40:01 UTC

On Wed, Sep 28, 2022 at 3:24 AM Dominique MARTINET

Post by Dominique MARTINET
...
Ugh, this is going to be a massive headache...
Other distributions I've worked with (e.g. nixos) have a wrapper for gcc
and clang that just enforce the flags they want the distro to be built
with -- I don't think debian has anything like that, would that be
easier to work with? My line of thinking is that there will only be a
single place to fix instead of configure/cmake/meson and all hand-made
build scripts that exist around here.

Debian has dpkg-buildflags . Discussions about it show up when
discussing hardening, like [0].

[0] https://wiki.debian.org/Hardening#dpkg-buildflags

Jeff

Wookey

2022-09-29 00:20:01 UTC

Post by Jeffrey Walton
On Wed, Sep 28, 2022 at 3:24 AM Dominique MARTINET

Debian has dpkg-buildflags . Discussions about it show up when
discussing hardening, like [0].
[0] https://wiki.debian.org/Hardening#dpkg-buildflags

Right, and just changing it and rebuilding works very well. I did this
for PAC (pointer authentication) support last year. Very few packages
do not correctly honour dpkg-buildflags.

In fact the only issue was that I unconditionally changed it so it got
issued by some packages (like migw) for the wrong-arch compiler
(because they were cross-building). One should be a bit smarter to
unconditionally set an _arch-specific_ flag.

dpkg-buildflags has functionality for this. See patch at bottom of:
https://lists.debian.org/debian-dpkg/2022/05/msg00022.html

Presumably the 'use 64-bit time_t' flag is the same for all arches,
but may only exist on 32-bit arches?

Wookey

--
Principal hats: Debian, Wookware, ARM
http://wookware.org/

Arnd Bergmann

2022-09-29 07:50:01 UTC

Post by Jeffrey Walton
On Wed, Sep 28, 2022 at 3:24 AM Dominique MARTINET

Debian has dpkg-buildflags . Discussions about it show up when
discussing hardening, like [0].
[0] https://wiki.debian.org/Hardening#dpkg-buildflags

I think the problem with using dpkg-buildflags is that it breaks
any user building their own applications against Debian provided
libraries, unless they remember to set the flag manually.

Post by Wookey
https://lists.debian.org/debian-dpkg/2022/05/msg00022.html
Presumably the 'use 64-bit time_t' flag is the same for all arches,
but may only exist on 32-bit arches?

Yes. The flag is -D_TIME_BITS=64, and this can be set unconditionally
everywhere if that helps. Note that this implies -D_FILE_OFFSET_BITS=64,
i.e. you can't have 64-bit time_t without 64-bit off_t/ino_t/...

As far as I understand, there are still a few libraries in Debian
that are built with 32-bit off_t in order to not break the traditional
ABI, and that is fine because off_t is rarely used in non-libc
header files, and a lot of applications don't actually need to
understand large file sizes (e.g. when they call 'fstat()' to
get the file timestamp but not the size, or when they operate
on files that are otherwise limited to less than 4GB).

So with time64, there will be an ABI change both for the few
libraries that are currently using 32-bit off_t in addition to
those that use time_t/timeval/timespec, and the original problem
with off_t is amplified because there are many more libraries
that have ABIs based around time_t.

This is also going to be hard for non-C languages with C bindings
(python, rust, ...) that have structure types hardcoded.

Arnd

Wookey

2022-09-29 10:00:02 UTC

Post by Arnd Bergmann
I think the problem with using dpkg-buildflags is that it breaks
any user building their own applications against Debian provided
libraries, unless they remember to set the flag manually.

Yes, but I'm talking about how to do a test rebuild. We set the
dpkg-buildflags config to always use -D_TIME_BITS=64 in the build
chroot, and build stuff.

For the actual new arch we woud set this as a gcc default for the
architecture as it's a significant part of the ABI. (and someone would
have to run gcc with -D_TIME_BITS=32 to explicitly build something for
the old abi (or use the <armhf-triplet> gcc).

Post by Arnd Bergmann
The flag is -D_TIME_BITS=64, and this can be set unconditionally
everywhere if that helps. Note that this implies -D_FILE_OFFSET_BITS=64,
i.e. you can't have 64-bit time_t without 64-bit off_t/ino_t/...
As far as I understand, there are still a few libraries in Debian
that are built with 32-bit off_t in order to not break the traditional
ABI,
So with time64, there will be an ABI change both for the few
libraries that are currently using 32-bit off_t in addition to
those that use time_t/timeval/timespec
This is also going to be hard for non-C languages with C bindings
(python, rust, ...) that have structure types hardcoded.

Hmm. Yes that could be interesting.

Are you aware of anyone else having written up efforts around this
(e.g. if the arch people have changed it already they presumably found
a load of the things one might trip over).

Wookey

--
Principal hats: Debian, Wookware, ARM
http://wookware.org/

Arnd Bergmann

2022-09-29 10:30:01 UTC

Yes, but I'm talking about how to do a test rebuild. We set the
dpkg-buildflags config to always use -D_TIME_BITS=64 in the build
chroot, and build stuff.

Right, that should work. In my 2020 experiment I did the opposite
and actually patched glibc to remove the time32 interfaces so
I could be really sure that everything would use the time64
path, but clearly at least the core packages all use the
buildflags correctly.

Post by Wookey
Hmm. Yes that could be interesting.
Are you aware of anyone else having written up efforts around this
(e.g. if the arch people have changed it already they presumably found
a load of the things one might trip over).

Adelie Linux has (had?) a great summary. I can't reach the wiki page
at the moment, but I found an archive copy at

https://web.archive.org/web/20220301175235/https://wiki.adelielinux.org/wiki/Project:Time64

Arnd

Wookey

2022-09-29 10:40:01 UTC

Yes, but I'm talking about how to do a test rebuild. We set the
dpkg-buildflags config to always use -D_TIME_BITS=64 in the build
chroot, and build stuff.

That is more reliable. We will keep the build logs so can check those
with just grep, or more smartly with a modified blhc, for anything
that did not in fact get the flag set. If it proves to be a problem
then a patched glibc is obviously a sensible approach.

Post by Wookey
Are you aware of anyone else having written up efforts around this
(e.g. if the arch people have changed it already they presumably found
a load of the things one might trip over).

Adelie Linux has (had?) a great summary. I can't reach the wiki page
at the moment, but I found an archive copy at
https://web.archive.org/web/20220301175235/https://wiki.adelielinux.org/wiki/Project:Time64

Cheers.

Wookey

--
Principal hats: Debian, Wookware, ARM
http://wookware.org/

Wookey

2022-09-29 00:10:01 UTC

Post by Dominique MARTINET
As far as I'm aware, there haven't been any new public discussion since
that 2020 thread

I mentioned it in the 'arm status' talk at debconf this year.

-- has there been any new attempt I missed?

I have been planning for a while to have a go at a bootstrap but have
not actually done anything yet. It is rising up my list as 'most
important task' (currently after 'fixing porterbox abel').

Post by Dominique MARTINET
I won't bring anything very useful to the table here: I'd be happy to
give some limited time,

I'm glad to hear that there is some interest from others.

We don't have a mechanism for a flag day SFAIK. We can't suddenly
re-upload every package in the armhf archive. And people would still
download new 64bit time_t packages into their existing armhf
installations and (presumably) have everything break. It's a new ABI -
we have to assume the breakage would be pretty bad.

It would be nice to do it within the armhf archive but although we
have mechanisms for core library transitions (like we did for libc5 to
6) a) this affects all arches so is a lot of churn for an issue only
affecting one arch (or maybe all 32-bit arches, but I'm not sure how
many of those other than arm (v7) will be around much longer?

Given the nature of this change I'm not even sure this would work if
we were up for the work, and I don't think we collectively are.

Post by Dominique MARTINET
So that might be the first thing to think about again?

A new debian arm ABI arch is definitely the easiest way to do this, as
Arndt says. Having thought about it a bit, I suggest we just call
it 'arm32' as in the future it will be useful to distinguish legacy
32-bit arm from arm64 (and maybe armv9 or whatever variants come in the
future). And having just 'arm32' and 'arm64' arches will be quite
pleasing for however long it lasts.

Obviously bikeshedding about the name and triplet is the least important part of this.

Post by Dominique MARTINET
Last time, the discussion got stuck because that means introducing a
new target name for dpkg, which then requires a new target triplet
to allow a multiarch installation. This in turn requires additional
work for porting packages that need to know about the new target
triplet. I don't think we can solve this problem now, but should
postpone the inevitable debate about this until someone has done the
work of rebootstrapping a working armhf environment with the current
names.

Agreed. And this was my plan. Just so we can check that stuff
basically works before patching toolchain builds. Given that arch have
it working it should be (?) as simple as setting some flags and
doing a rebuild (and fixing whatever breaks).

Wookey
--
Principal hats: Debian, Wookware, ARM
http://wookware.org/

Arnd Bergmann

2022-09-29 08:20:01 UTC

Post by Dominique MARTINET
As far as I'm aware, there haven't been any new public discussion since
that 2020 thread

I mentioned it in the 'arm status' talk at debconf this year.
-- has there been any new attempt I missed?
I have been planning for a while to have a go at a bootstrap but have
not actually done anything yet. It is rising up my list as 'most
important task' (currently after 'fixing porterbox abel').

Ah nice! Once you actually get to it, feel free to contact me
in private if you would like me to walk you through what I did
last time, and what problems I had.

Post by Wookey
It would be nice to do it within the armhf archive but although we
have mechanisms for core library transitions (like we did for libc5 to
6) a) this affects all arches so is a lot of churn for an issue only
affecting one arch (or maybe all 32-bit arches, but I'm not sure how
many of those other than arm (v7) will be around much longer?

Right, I think it's fairly safe to assume that the other three (i386,
mipsel and armel) will be discontinued well before armhf, and early
enough that they are not affected too badly by the y2038 overflow.

There is a chance that there is a use case for having a minimal
i386-y2038 target for the purpose of running proprietary 32-bit
software (i.e. windows games using wine), but that can come later.

Post by Dominique MARTINET
So that might be the first thing to think about again?

A new debian arm ABI arch is definitely the easiest way to do this, as
Arndt says. Having thought about it a bit, I suggest we just call
it 'arm32' as in the future it will be useful to distinguish legacy
32-bit arm from arm64 (and maybe armv9 or whatever variants come in the
future). And having just 'arm32' and 'arm64' arches will be quite
pleasing for however long it lasts.
Obviously bikeshedding about the name and triplet is the least
important part of this.

I was not aware of Arch linux using a time64 build. The main ones
I know are Alpine Linux, Adelie Linux and OpenWRT, but those all
use musl-1.2, not glibc, and they have a much smaller set of packages.
https://wiki.musl-libc.org/projects-using-musl.html lists a few
other distros that I'm not familiar with.

Arnd

Wookey

2022-09-29 10:00:01 UTC

Post by Wookey
Given that arch have
it working it should be (?) as simple as setting some flags and
doing a rebuild (and fixing whatever breaks).

I was not aware of Arch linux using a time64 build.

Sorry - that's me not reading properly. You wrote 'Alpine' and my
brain read 'Arch'. I have just repeated the error in my last mail.

The main ones

Post by Arnd Bergmann
I know are Alpine Linux, Adelie Linux and OpenWRT, but those all
use musl-1.2, not glibc, and they have a much smaller set of packages.

OK. We still have plenty of bugs to find then :-)

Wookey

--
Principal hats: Debian, Wookware, ARM
http://wookware.org/

Adrian Bunk

2022-10-02 12:20:01 UTC

Post by Wookey
...

Post by Arnd Bergmann
I know are Alpine Linux, Adelie Linux and OpenWRT, but those all
use musl-1.2, not glibc, and they have a much smaller set of packages.

OK. We still have plenty of bugs to find then :-)

Such bugs have already been found, reported and fixed in Debian for
10 years thanks to the x32 port (e.g. #700012).

More than 13k packages are currently Installed on x32 after having been
natively built and (if they have) passing their build time tests.

Post by Wookey
Wookey

cu
Adrian

Arnd Bergmann

2022-10-02 14:40:01 UTC

Post by Adrian Bunk

Post by Wookey
...

Post by Arnd Bergmann
I know are Alpine Linux, Adelie Linux and OpenWRT, but those all
use musl-1.2, not glibc, and they have a much smaller set of packages.

OK. We still have plenty of bugs to find then :-)

Such bugs have already been found, reported and fixed in Debian for
10 years thanks to the x32 port (e.g. #700012).
More than 13k packages are currently Installed on x32 after having been
natively built and (if they have) passing their build time tests.

There is a minimal user base on x32, so I'm sure a lot of bugs
have gone unnoticed. Having 64-bit time_t on BSD and Windows
probably did more to find bugs, but there are still lots of
issues where neither of this would help. For instance:

- Any applications that use direct system calls with a 32-bit
time_t have to change to the correct replacement syscall.
This never had to be done on x32, because there is only one
such set of syscalls. A number of upstream packages already
got fixed for riscv32, but unfortunately some of those were
done in a way that is still broken for architectures that
have both the time32 and time64 versions.
Most commonly, this affects __NR_futex/SYS_futex, but there
are other syscalls needed elsewhere

- Libraries using input_event structures on /dev/input/*
need to have an updated copy of the kernel headers. Most
upstream packages do now, but I'm sure there are some left.

- Packages that rely on seccomp have to be updated to
allow both the old and new versions of system calls in their
whitelists

- Anything that is written in a language other than C but
links to C libraries needs to be updated to use the new
data structure definitions. Some of these may have a special
case for x32, or they are just wrong. There are a lot of
python or rust libraries affected by this, and no obvious
answer.

- Things that are written in a language other than C/C++
but don't link to C libraries should keep working, but
will not be y2038 safe unless they also migrate to the
time64 interfaces by copying what glibc did.

- Any package that currently relies on having a 32-bit
off_t/ino_t will break after being forced to update to
the 64-bit version, even if they don't care about
time_t.

- Packages may hardcode the time_t/timeval/timespec
definition. If they use __kernel_long_t, they would
even work on x32, but break on any other 32-bit ABI.

Arnd

Adrian Bunk

2022-10-02 16:40:01 UTC

Post by Adrian Bunk

Post by Wookey
...

Post by Arnd Bergmann
I know are Alpine Linux, Adelie Linux and OpenWRT, but those all
use musl-1.2, not glibc, and they have a much smaller set of packages.

OK. We still have plenty of bugs to find then :-)

Such bugs have already been found, reported and fixed in Debian for
10 years thanks to the x32 port (e.g. #700012).
More than 13k packages are currently Installed on x32 after having been
natively built and (if they have) passing their build time tests.

There is a minimal user base on x32, so I'm sure a lot of bugs
have gone unnoticed. Having 64-bit time_t on BSD and Windows
probably did more to find bugs, but there are still lots of
...

I am not denying that there will be bugs to be sorted out,
but with x32 (and likely soon arc) in ports many of the issues
have already been sorted out and there are also build logs that
could be searched for time_t gcc warnings in successful builds.

Post by Arnd Bergmann
- Things that are written in a language other than C/C++
but don't link to C libraries should keep working,

Being written in C and linking to C libraries is not sufficient, we have
several other C libraries.

E.g. dietlibc:
https://sources.debian.org/src/dietlibc/0.34~cvs20160606-14/include/sys/types.h/#L106-L110

Post by Arnd Bergmann
but
will not be y2038 safe unless they also migrate to the
time64 interfaces by copying what glibc did.
...

Yes, code breaking on 19 January 2038 on the fixed architectures should
be a big worry.

The other breakages you describe are things that would break immediately.
Most would be found during the years before the new port would be part
of a Debian release, and users would encounter remaining ones when
switching to the new architecture at a time that is convenient for them
(ideally first in a test environment).

Running test builds (as reproducible builds is already doing) and
autopkgtests with time set in the future might catch bugs that are
covered by tests, but would still miss many.

Post by Arnd Bergmann
...
Last time, the discussion got stuck because that means introducing
a new target name for dpkg, which then requires a new target triplet
to allow a multiarch installation. This in turn requires additional
work for porting packages that need to know about the new target
triplet. I don't think we can solve this problem now, but should
postpone the inevitable debate about this until someone has done
the work of rebootstrapping a working armhf environment with the
current names.
...

IMHO there is little point in spending work anywhere on this unless
there is a commitment and schedule for the whole big amount of work
of setting up the new port.

The port would need all of:
- finding bugs
- fixing bugs
- adding the new architecture to debian/ in ~ 1k packages

And all this with a schedule to have it in a releasable state by the end
of 2024 or 2026 if it would start today.

Without such a schedule, other distributions will be faster then Debian
at switching to 64bit time_t, already resolving many of the issues you
are planning to find with the rebootstrap.

Having the new port in a releasable state in 2032, with a Debian release
with both old and new port in mid-2033, might be the last possible
schedule for Debian.

cu
Adrian

Wookey

2023-02-01 05:20:01 UTC