another attempt at Y2038

[Helmut Grohne]

Hi,

I was also wondering about this Y2038 thingy and did some experiments.
I'm reporting what I found to document it, but don't see much actionable
stuff here. Many thanks to Arnd Bergmann for his input.

Attempt #1: rebootstrap

Given that I develop rebootstrap, attempting to use it for a time64
bootstrap seemed quite natural. I've been talking to this with Steve
multiple times including DC22. The question was how to plug it in. In
the end I went for Arnd's suggestion to set DPKG_*_APPEND variables to
modify dpkg-buildflags. Not every package uses these flags, but a
majority do. For a survey, this is probably good enough.

Things went somewhat far. The first issue was zlib. There isn't much to
say about it: https://github.com/madler/zlib/issues/447

Then the libprelude build failed with symbol issues. It happens to build
a C++ library and its symbols are dependent on time types. Unsurprising.
This is a case where we do break ABI.

Then it occured to me that not failing does not imply not being affected
right? You can break ABI without failing the build. So as a detection
method this has a significant risk of false negatives.

For just bootstrapping an incompatible time64 armhf, this method should
mostly just work. Would someone like to have the resulting .debs? What
would do with them?

Attempt #2: reproducible

My other favourite QA tool is reproducible builds. So how about building
and comparing binary packages?

Unfortunately, build flags participate in build ids. That directly
implies a high risk of false positives and/or manual work.

Without further ado, here we go.

.debs unchanged:
* base-files
* db-defaults
* coreutils
* diffutils
* findutils
* libonig
* libidn
* libidn2
* libpipeline
* libpthread-stubs
* libsm
* gzip

Question: How many of these ignore dpkg-buildflags?

minor changes:
* gmp: build-id and #define __GMP_CFLAGS
* guile-3.0: build-id and recorded build path

unclear:
* fribidi: only libfribidi-bin changes, probably harmless

ftbfs:
* zlib: see earlier

starts using symbols such as __stat64_time64, __ioctl_time64:
* acl
* attr
* base-passwd
* bash
* blt
* bsdmainutils
* bzip2
* c-ares
* curl
* dash
* debianutils
* dpkg
* e2fsprogs
* expat
* file
* flex
* fontconfig
* freetype
* fuse
* glib2.0
* gnupg2
* gpm
* groff
* hostname
* icu
* jansson
* jemalloc
* keyutils
* kmod
* krb5
* libassuan
* libatomic-ops
* libbsd
* libcap2
* libdebian-installer
* libev
* libevent
* libgc
* libgcrypt20
* libice
* libksba
* libmd
* libnsl
* libpng1.6
* libsepol

starts using large file symbols, but no time64 symbols:
* libffi

starts using symbols such as above, but also changes C++ symbols:
* apt

unreproducible for other non-time reasons:
* binutils

After this sample, it became clear that this method is also not very
helpful. While it shows that basically everything is affected, it
doesn't give a good handle on ABI breakage either.

Please Cc me in replies as I am not subscribed.

Helmut

[Arnd Bergmann]

Post by Helmut Grohne
Hi,
I was also wondering about this Y2038 thingy and did some experiments.
I'm reporting what I found to document it, but don't see much actionable
stuff here. Many thanks to Arnd Bergmann for his input.

Thanks a lot for taking a look here, this is very promising, compared
to my attempt from 2020.

Post by Helmut Grohne
Attempt #1: rebootstrap
Given that I develop rebootstrap, attempting to use it for a time64
bootstrap seemed quite natural. I've been talking to this with Steve
multiple times including DC22. The question was how to plug it in. In
the end I went for Arnd's suggestion to set DPKG_*_APPEND variables to
modify dpkg-buildflags. Not every package uses these flags, but a
majority do. For a survey, this is probably good enough.
Things went somewhat far. The first issue was zlib. There isn't much to
say about it: https://github.com/madler/zlib/issues/447
Then the libprelude build failed with symbol issues. It happens to build
a C++ library and its symbols are dependent on time types. Unsurprising.
This is a case where we do break ABI.

Just for my understanding: what would be the correct way to handle this,
under the assumption that the new symbol names that dpkg-shlibdeps
found are the ones we want? Is there a way to flag both the time32
and the tim64 set of symbols as correct, or would we have to have a
per-architecture list of expected symbols for the new (still to be
decided) architecture name?

Post by Helmut Grohne
Then it occured to me that not failing does not imply not being affected
right? You can break ABI without failing the build. So as a detection
method this has a significant risk of false negatives.
For just bootstrapping an incompatible time64 armhf, this method should
mostly just work. Would someone like to have the resulting .debs? What
would do with them?

I think Wookey wants to do a build of a much larger set of packages
as a feasibility study to show that a rebuild actually survives running
with time set to 2038+, before a decision is made about how and when
this becomes an official release target.

Post by Helmut Grohne
Attempt #2: reproducible
My other favourite QA tool is reproducible builds. So how about building
and comparing binary packages?
Unfortunately, build flags participate in build ids. That directly
implies a high risk of false positives and/or manual work.
Without further ado, here we go.
* base-files
* db-defaults
* coreutils
* diffutils
* findutils
* libonig
* libidn
* libidn2
* libpipeline
* libpthread-stubs
* libsm
* gzip
Question: How many of these ignore dpkg-buildflags?

coreutils definitely should have changed, given that 'ls'
prints timestamps.

Post by Helmut Grohne
After this sample, it became clear that this method is also not very
helpful. While it shows that basically everything is affected, it
doesn't give a good handle on ABI breakage either.

I think it's safe to say that most packages will be affected in
some way. Packages that only use glibc and no other libraries could
be converted separately, but this is not helpful in the end since it
does not get any closer to converting libraries that export
modified interfaces. As far as I remember, I could show (using
pattern matching on header files) that about half the libraries
in Debian will require have an ABI change, but a much higher number
of packages depend on at least one library aside from glibc that
changes like this. To actually get this done, everything must be
rebuilt, and in order to coexist with existing binaries, this
requires a new architecture name and target triplet.

Arnd

[Sune Vuorela]

Post by Arnd Bergmann
Just for my understanding: what would be the correct way to handle this,
under the assumption that the new symbol names that dpkg-shlibdeps
found are the ones we want? Is there a way to flag both the time32
and the tim64 set of symbols as correct, or would we have to have a
per-architecture list of expected symbols for the new (still to be
decided) architecture name?

This is pretty normal for c++ libraries to have either symbol files
depending on the architecture or symbol files or symbol somehow
templated and filled out at build time (e.g. pkgkde-symbolshelper)

/Sune

[John Paul Adrian Glaubitz]

Hi Helmut!

Post by Helmut Grohne
I was also wondering about this Y2038 thingy and did some experiments.
I'm reporting what I found to document it, but don't see much actionable
stuff here. Many thanks to Arnd Bergmann for his input.
Attempt #1: rebootstrap
Given that I develop rebootstrap, attempting to use it for a time64
bootstrap seemed quite natural. I've been talking to this with Steve
multiple times including DC22. The question was how to plug it in. In
the end I went for Arnd's suggestion to set DPKG_*_APPEND variables to
modify dpkg-buildflags. Not every package uses these flags, but a
majority do. For a survey, this is probably good enough.

I would love to do that for m68k as well. We could use this opportunity to
rebuild the m68k port with 32-bit alignment which would solve quite a number
of problems since many projects like LLVM and Qt assume a minimum alignment
of 32 bits while m68k still defaults to 16 bits.

Since the time64 rebootstrap would break the ABI anyway, we could use to fix
alignment issue on m68k once and for all :-).

Adrian

[Steve McIntyre]

Post by John Paul Adrian Glaubitz
Hi Helmut!

Post by Helmut Grohne
I was also wondering about this Y2038 thingy and did some experiments.
I'm reporting what I found to document it, but don't see much actionable
stuff here. Many thanks to Arnd Bergmann for his input.
Attempt #1: rebootstrap
Given that I develop rebootstrap, attempting to use it for a time64
bootstrap seemed quite natural. I've been talking to this with Steve
multiple times including DC22. The question was how to plug it in. In
the end I went for Arnd's suggestion to set DPKG_*_APPEND variables to
modify dpkg-buildflags. Not every package uses these flags, but a
majority do. For a survey, this is probably good enough.

I would love to do that for m68k as well. We could use this opportunity to
rebuild the m68k port with 32-bit alignment which would solve quite a number
of problems since many projects like LLVM and Qt assume a minimum alignment
of 32 bits while m68k still defaults to 16 bits.
Since the time64 rebootstrap would break the ABI anyway, we could use to fix
alignment issue on m68k once and for all :-).

So the reason that we're talking about doing a replacement armhf port
is for two reasons:

* it's probably one of the few 32-bit arches that is likely to still
be in routine use beyond 2038 (*many* embedded users depending on
Debian stuff)

* it's feasible to rebootstrap and break ABI as there isn't a large
corpus of older-ABI binaries that people will care about supporting
in the future. (This rules out i386 - the main reason to keep it
around is just for the older binaries AFAICS.)

I don't want to be rude, but I really don't see how m68k fits here. Of
course, feel free to do a rebootstrap if you like, but I genuinely
don't see any great need for it.

[John Paul Adrian Glaubitz]

Post by Steve McIntyre
I don't want to be rude, but I really don't see how m68k fits here. Of
course, feel free to do a rebootstrap if you like, but I genuinely
don't see any great need for it.

I don't see the need for starting another "Why does Debian Ports exist?"
discussion again. If you want to discuss about what's needed, probably,
a lot of open source projects would just have to go away. ¯\_(ツ)_/¯

People just enjoy hacking code. There doesn't always have to have a commercial
interest and a bigger reason behind it and that's perfectly fine.

Adrian

[Thomas Schmitt]

Hi,

likely to still be in routine use beyond 2038

Sidenote towards ISO 9660 image producers:

Don't forget to check from time to time whether Linux removed the
int bottleneck in fs/isofs/.
See:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800627#38

The bug report was originally about a Y2028 rollover caused by 8-bit
signedness. It was closed after
https://github.com/torvalds/linux/commit/34be4dbf87fc
which left the int problem for future adventures.

Currently it's still "int iso_date()" in:
https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h line 109
https://github.com/torvalds/linux/blob/master/fs/isofs/util.c line 19


Have a nice day :)

Thomas

[Arnd Bergmann]

Post by Thomas Schmitt
Hi,

likely to still be in routine use beyond 2038

Don't forget to check from time to time whether Linux removed the
int bottleneck in fs/isofs/.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800627#38
The bug report was originally about a Y2028 rollover caused by 8-bit
signedness. It was closed after
https://github.com/torvalds/linux/commit/34be4dbf87fc
which left the int problem for future adventures.
https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h line 109
https://github.com/torvalds/linux/blob/master/fs/isofs/util.c line 19
Have a nice day :)

That's just a trivially fixed bug, right? I don't recall
ever seeing a bug report or a patch for it in the past. Would
you like to send the patch, or should I?

Arnd

[Arnd Bergmann]

Post by Arnd Bergmann

Post by Thomas Schmitt
https://github.com/torvalds/linux/blob/master/fs/isofs/isofs.h line 109
https://github.com/torvalds/linux/blob/master/fs/isofs/util.c line 19
Have a nice day :)

That's just a trivially fixed bug, right? I don't recall
ever seeing a bug report or a patch for it in the past. Would
you like to send the patch, or should I?

I see you actually put the patch into the debian bug
tracker, though I can't find it on lore.kernel.org. Where
did you submit it?

Arnd

[Thomas Schmitt]

Hi,

Post by Arnd Bergmann
That's just a trivially fixed bug, right?

Yes. Very simple. Just s/int/time64_t/ .

Post by Arnd Bergmann
I don't recall ever seeing a bug report or a patch for it in the past.

I did not submit it to LKML or linux-scsi, because i got no reply to
earlier attempts to bring in my userland knowledge about ISO 9660 and
optical drives:

https://lore.kernel.org/linux-scsi/20201120140633.1673-1-***@gmx.net/T/
[PATCH] isofs: fix Oops with zisofs and large PAGE_SIZE

https://lore.kernel.org/linux-scsi/20201006094026.1730-1-***@gmx.net/T/
[PATCH v2 0/2] Fix automatic CD tray loading for data reading via sr
[PATCH v2 1/2] cdrom: delegate automatic CD tray loading to callers of
cdrom_open()
[PATCH v2 2/2] sr: fix automatic tray loading for data reading

I made more bug fixes and a wishlist patch two years ago.
But keeping them up to date with the agile kernel development is quite a
big task for me. (As said: userlander.)

Especially fs/isofs has no maintainer, so i could only submit to linux-scsi
because of the proximity to cdrom and sr. I had hoped that above two
patches would be considered as modest self-introduction, but obviously my
social skills are not sufficient.

kernelnewbies could not help me either. I am still subscribed there in
the hope to find allies. At one occasion i saw a chance when Adverg
Ebashinskii asked for low hanging fruit in fs:
https://www.mail-archive.com/search?q=Kernel+bug+tracker&l=kernelnewbies%40kernelnewbies.org
I proposed the Y2038 patch:
https://www.mail-archive.com/***@kernelnewbies.org/msg21629.html
with the offer to reduce my "Signed-off-by:" to "Suggested-by:".

This is what i think needed mending in kernel 5.10 two years ago:

[PATCH 0/3] Fix the old CD read-ahead bug for media with a single TAO track
(Most drives report the 2 unreadable TAO run-out blocks as
part of the medium capacity.)

[PATCH 0/1] sr: verify that last_written block is readable before deriving
size from it
(Size assessment of optical media in Linux is quite a mess
and can overshoot beyond the TAO run-out problem.)

[PATCH 0/4] Attribute size 0 to sr device if no readable medium is loaded
(Linux reports the size of the last loaded medium, or 2048
bytes if a blank medium is inserted.)

[PATCH 0/4] Make mount -t iso9660 -o session=N work on DVD and BD media up
to N=168
(While -o sbsector= works for all media types, session= works
only with CD media.)

[PATCH 0/1] isofs: truncate oversized Rock Ridge names to 255 bytes
(Truncation currently happens if name length is >= 254 and
then cuts off much more of the name than needed.)

When those bugs would be fixed (or mitigated in case of TAO), i hoped to
get a favor for my own hobby:

[PATCH 0/3] Introduce a new ioctl CDROM_SIMUL_CHANGE for burn programs
(Currently Linux knows about new content created via ioctl
SG_IO only after eject and reload of the Medium.)

The housekeeping aspects of kernel development are really hard for me to
master. I don't strive to become a regular contributor. But seeing those
bugs since years causes me to mention them when there is hope to meet
kernel developers.

So:
Thanks a lot for replying. Is there a chance to get you interested in the
other bugs above and maybe even my wish for ioctl CDROM_SIMUL_CHANGE ?


Have a nice day :)

Thomas

[Arnd Bergmann]

Post by Thomas Schmitt
[PATCH] isofs: fix Oops with zisofs and large PAGE_SIZE
[PATCH v2 0/2] Fix automatic CD tray loading for data reading via sr
[PATCH v2 1/2] cdrom: delegate automatic CD tray loading to callers of
cdrom_open()
[PATCH v2 2/2] sr: fix automatic tray loading for data reading
I made more bug fixes and a wishlist patch two years ago.
But keeping them up to date with the agile kernel development is quite a
big task for me. (As said: userlander.)

These look like you did everything right, and they should have been
picked up by the scsi maintainers. If you ever re-send them, I would
suggest putting the maintainers in 'To:' for the email and the mailing
list in Cc, and asking them specifically to pick up the patches.

Post by Thomas Schmitt
Especially fs/isofs has no maintainer, so i could only submit to linux-scsi
because of the proximity to cdrom and sr. I had hoped that above two
patches would be considered as modest self-introduction, but obviously my
social skills are not sufficient.

I think the hard part here is knowing who to send the patches to.
Unmaintained file systems are particularly tricky, in this case I
would have used

To: Alexander Viro <***@zeniv.linux.org.uk>
To: Jan Kara <***@suse.cz>
To: Andrew Morton <***@linux-foundation.org>
Cc: Arnd Bergmann <***@arndb.de>
Cc: Deepa Dinamani <***@gmail.com>
Cc: linux-***@vger.kernel.org
Cc: linux-***@vger.kernel.org
Cc: ***@lists.linaro.org

Can you rebase the patch on top of v6.1-rc1 and send it to
this list of people? I'll reply with a 'Reviewed-by' tag,
and one of the above should be able to pick it up.

Post by Thomas Schmitt
[PATCH 0/3] Fix the old CD read-ahead bug for media with a single TAO track
(Most drives report the 2 unreadable TAO run-out blocks as
part of the medium capacity.)
[PATCH 0/1] sr: verify that last_written block is readable before deriving
size from it
(Size assessment of optical media in Linux is quite a mess
and can overshoot beyond the TAO run-out problem.)
[PATCH 0/4] Attribute size 0 to sr device if no readable medium is loaded
(Linux reports the size of the last loaded medium, or 2048
bytes if a blank medium is inserted.)
[PATCH 0/4] Make mount -t iso9660 -o session=N work on DVD and BD media up
to N=168
(While -o sbsector= works for all media types, session= works
only with CD media.)
[PATCH 0/1] isofs: truncate oversized Rock Ridge names to 255 bytes
(Truncation currently happens if name length is >= 254 and
then cuts off much more of the name than needed.)
When those bugs would be fixed (or mitigated in case of TAO), i hoped to
[PATCH 0/3] Introduce a new ioctl CDROM_SIMUL_CHANGE for burn programs
(Currently Linux knows about new content created via ioctl
SG_IO only after eject and reload of the Medium.)
The housekeeping aspects of kernel development are really hard for me to
master. I don't strive to become a regular contributor. But seeing those
bugs since years causes me to mention them when there is hope to meet
kernel developers.
Thanks a lot for replying. Is there a chance to get you interested in the
other bugs above and maybe even my wish for ioctl CDROM_SIMUL_CHANGE ?

I mainly care about the y2038 issue here, haven't done anything interesting
on cdrom drivers in 20 years ;-)

Arnd

[Thomas Schmitt]

Hi,

Post by Arnd Bergmann
These look like you did everything right, and they should have been
picked up by the scsi maintainers.

I thought the same, but also that insisting, screaming, or trampling
won't help.

Post by Arnd Bergmann
If you ever re-send them, I would
suggest putting the maintainers in 'To:' for the email and the mailing
list in Cc,

I remember to have looked into the MAINTAINERS file and the git log for
names.
(In the case of the powerpc64 Oops, it looks like i added you in Cc:.
If i only could remember why ...)

Post by Arnd Bergmann
I think the hard part here is knowing who to send the patches to.
Unmaintained file systems are particularly tricky, in this case I
would have used
Can you rebase the patch on top of v6.1-rc1 and send it to
this list of people?

I know the word "rebase" but cannot promise that i can fill it with
substance soon ...

What kernel branches should i choose for sr and for isofs ?

Post by Arnd Bergmann
I mainly care about the y2038 issue here,

If you want to do us both a favor then bring the changes from
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800627#38
into the kernel.
Feel free to take my reasoning and demonstration text. The code change is
trivial. So if at all, i would be fully recognized by "Suggested-by:".

It might be worth to verify my claims:

The only callers of iso_date() are in isofs/inode.c and isofs/rock.c
and put the result into struct inode.i_{a,c,m}time.tv_sec which is
of type time64_t.
The time value of iso_date() essentially stems from mktime64().

and to exercise the demonstration by a xorriso-made ISO.


Have a nice day :)

Thomas

[Arnd Bergmann]

Post by Thomas Schmitt

Post by Arnd Bergmann
I think the hard part here is knowing who to send the patches to.
Unmaintained file systems are particularly tricky, in this case I
would have used

This would be the correct list for the cdrom driver patches,
my list above would be for the isofs time64 patch.

Post by Thomas Schmitt

Post by Arnd Bergmann
Can you rebase the patch on top of v6.1-rc1 and send it to
this list of people?

I know the word "rebase" but cannot promise that i can fill it with
substance soon ...
What kernel branches should i choose for sr and for isofs ?

It's generally ok to just use the latest -rc1 (right now 6.1-rc1)
as the base, unless a maintainer asks you to use their tree
as a base.

Post by Thomas Schmitt

Post by Arnd Bergmann
I mainly care about the y2038 issue here,

If you want to do us both a favor then bring the changes from
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800627#38
into the kernel.
Feel free to take my reasoning and demonstration text. The code change is
trivial.

Done now. I've left you as author and first Signed-off-by
though. While the change itself is trivial, the important bit
is identifying the problem, and you did that.

Post by Thomas Schmitt
The only callers of iso_date() are in isofs/inode.c and isofs/rock.c
and put the result into struct inode.i_{a,c,m}time.tv_sec which is
of type time64_t.
The time value of iso_date() essentially stems from mktime64().
and to exercise the demonstration by a xorriso-made ISO.

I could immediately tell that your patch is correct when I saw it.

Arnd

[Thomas Schmitt]

Hi,

Post by Arnd Bergmann
This would be the correct list for the cdrom driver patches,
my list above would be for the isofs time64 patch.

Good to know. I'll keep both lists in my cheat sheet.
(I guess that ***@lists.linaro.org is not appropriate for non-Y2038
patches.)

Post by Arnd Bergmann
It's generally ok to just use the latest -rc1 (right now 6.1-rc1)

Understood. (Two minutes after i sent my latest mail i got yours.)

Post by Arnd Bergmann

Post by Thomas Schmitt
If you want to do us both a favor then bring the changes from
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800627#38
into the kernel.

Done now.

Thank you.

Also thanks to the bystanders for their tolerance.
At least my hijacking of this thread was a successful attempt at Y2038.


Have a nice day :)

Thomas

[Thomas Schmitt]

Hi,

Post by Thomas Schmitt

Post by Arnd Bergmann
Can you rebase the patch on top of v6.1-rc1

What kernel branches should i choose for sr and for isofs ?

Ok, that question was probably stupid.

May i understand your advise as
"on top of the newest vX.Y-rcZ in github.com/torvalds/linux"
?
(Given that my rebase attempts will not happen quite now.)


Have a nice day :)

Thomas

[Steve Langasek]

Hi Helmut, all,

Post by Helmut Grohne
Then the libprelude build failed with symbol issues. It happens to build
a C++ library and its symbols are dependent on time types. Unsurprising.
This is a case where we do break ABI.
Then it occured to me that not failing does not imply not being affected
right? You can break ABI without failing the build. So as a detection
method this has a significant risk of false negatives.
For just bootstrapping an incompatible time64 armhf, this method should
mostly just work. Would someone like to have the resulting .debs? What
would do with them?

This is good. One thing that I think has been missing from the discussion
about armhf rebootstrap is the fact that we do have experience in Debian
doing cross-cutting ABI transitions without having to change an
architecture name. We've done this at least three times in Debian history:
the libc5->libc6 transition ('g' suffix - which still remains today in
libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long
double' transition ('ldbl' suffix, and an example of doing this for an ABI
transition that didn't affect all architectures).

The first of these didn't require analysis, we knew all shared libraries
were affected (and Debian was much smaller then).

The second required analysis but it was easy because the C++ ABI is very
visible in ELF symbols.

The third is the best analogue to the time_t situation, because it involved
changing the size of a type that could appear not just in ABIs of C++
libraries, but also C libraries.

And we did manage to survive that intact!
https://lists.debian.org/debian-devel/2007/05/msg01173.html


Nowadays, I wonder if abi-compliance-checker would be usable for analyzing
the ABIs of C libraries to accurately identify what needs to change for
time64_t. I think it would be interesting to know from this how many shared
libraries expose time_t size in their ABIs.

[Steve Langasek]

Post by Steve Langasek
This is good. One thing that I think has been missing from the discussion
about armhf rebootstrap is the fact that we do have experience in Debian
doing cross-cutting ABI transitions without having to change an
the libc5->libc6 transition ('g' suffix - which still remains today in
libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long
double' transition ('ldbl' suffix, and an example of doing this for an ABI
transition that didn't affect all architectures).
The first of these didn't require analysis, we knew all shared libraries
were affected (and Debian was much smaller then).
The second required analysis but it was easy because the C++ ABI is very
visible in ELF symbols.
The third is the best analogue to the time_t situation, because it involved
changing the size of a type that could appear not just in ABIs of C++
libraries, but also C libraries.
And we did manage to survive that intact!
https://lists.debian.org/debian-devel/2007/05/msg01173.html
Nowadays, I wonder if abi-compliance-checker would be usable for analyzing
the ABIs of C libraries to accurately identify what needs to change for
time64_t. I think it would be interesting to know from this how many shared
libraries expose time_t size in their ABIs.

Well, I didn't see anybody else expressing interest in this, so I had a go.

I've done a first-pass analysis of Ubuntu lunar/armhf using
abi-compliance-checker. The results of my investigation, including scripts
and detailed reports/logs, are here:

https://people.canonical.com/~vorlon/armhf-time_t/

Notes:

- abi-compliance-checker was used to analyze headers of 767 library
packages in the archive. Of those, 82 of 558 were successfully analyzed
and identified as library packages that would need patched for the
switch to 64-bit time_t to mark them as ABI-incompatible with the
previous version on armhf.

- an additional 209 packages could not be analyzed, because a-c-c failed
to compile the headers using its invocation of gcc. Assuming a roughly
equal distribution of ABI-changing vs non-ABI-changing libraries among
these that have failed to compile, we'd be looking at around 113 of 767
libraries that need changed.

- if we decide the level of effort to patch this many libraries is
worthwhile, to get an actual actionable list of libraries that need
patched for this transition would require getting the compile failures
down to zero. I did dig into the failures alphabetically from "a" to
"libe"; most of these failures were resolved by one of three methods.
- letting a-c-c know that this is a C library and to not try to compile
it with gcc's C++ frontend;
- excluding various headers that can't be included directly, either
because they are internal headers that are included via other headers
in the package, or because they have dependencies on other headers not
in the archive and can't be compiled (and therefore can't affect the
library ABI);
- pulling in additional undeclared package dependencies.
Even when applying these fixes, I still had 8 library packages whose
headers I wasn't able to get to compile. So there's still quite a bit
of work to do here.

- On this preliminary pass, I only included library packages that shipped
both headers and .so's, as this supposedly gives the best quality
results from a-c-c. To get a full list of ABI-breaking changes,
however, we need to include packages that ship only headers and not
.so's, to catch those libraries for which the .so is in a different
binary package (boost), and to catch plugin systems for executables
(apache).

While I'm currently working on this using Ubuntu, the results are largely
applicable to Debian, and the script itself certainly is. Note however that
there are a couple of bugs in abi-compliance-checker that I've patched in
Ubuntu, bug #951076 and bug #1030540, which you would want to have fixed
before trying to run this analysis in Debian. Cc:ing the a-c-c maintainer
for awareness of this.

Does doing an ABI transition of ~113 libraries seem tractable to folks? If
there's interest in moving forward on this, I'd say the next step should be
to take it to debian-devel for broader discussion/signoff. I would also
move my scripts into a git repository that folks can contribute to -- adding
the necessary overrides for a-c-c for each library, so we can get an
authoritative list of ABI breakages, is very parallelizable.

Cheers,

[Wookey]

Post by Steve Langasek

Post by Steve Langasek
This is good. One thing that I think has been missing from the discussion
about armhf rebootstrap is the fact that we do have experience in Debian
doing cross-cutting ABI transitions without having to change an
the libc5->libc6 transition ('g' suffix - which still remains today in
libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long
double' transition ('ldbl' suffix, and an example of doing this for an ABI
transition that didn't affect all architectures).

I feel like I should already know this, but after a bit of thought,
and reading the transitions wiki page:
https://wiki.debian.org/Teams/ReleaseTeam/Transitions I do not
understand how the suffixes are used in these transitions.

That wiki page does not appear to mention adding suffixes to package
names. It appears to only document the process where the version is
just bumped, everything involved is rebuilt in experimental and then
re-uploaded to unstable once we are ready to make the change.

I presume that the suffix is used for transitions involving a large
number of packages, in order to keep both the old-ABI and new-ABI
version of the packages around in parallel (and conflicting with each
other?) for a while which is necessary for larger transitions because
doing it all in one go would take too long, or get entangled with
uploads from the unware that might cause breakage, or something?

Can someone point at an explanation of the process, or add it to the above page?

Just rebuilding and not renaming sounds like a much smoother process,
because as soon as we rename then there is a load of renaming in
reverse dependencies too, and there is presumably a set of rules about
when conflicts are added and other details that need to be got right.

I presume there is a good reason for this distinction between 'minor'
transitions and 'mega' transitions. I'd like to understand it properly.

Post by Steve Langasek

Post by Steve Langasek
Nowadays, I wonder if abi-compliance-checker would be usable for analyzing
the ABIs of C libraries to accurately identify what needs to change for
time64_t. I think it would be interesting to know from this how many shared
libraries expose time_t size in their ABIs.

Well, I didn't see anybody else expressing interest in this, so I had a go.
I've done a first-pass analysis of Ubuntu lunar/armhf using
abi-compliance-checker. The results of my investigation, including scripts

Thanks very much for this. Really helpful. I tried to use abi-dumper
and abi-compliance-checker but wasn't having much success in getting
useful reports out of them. I found abigail-tools to be quite a lot
simpler to use. Probably we should run both and investigat if they
disagree about differences.

I found out about your effort just in time to include it in my FOSDEM talk, which was good.

https://fosdem.org/2023/schedule/event/fixing_2038/ for anyone who is
unaware. (The Q&A has some dropouts and didn't add all that much
useful info so you can stop at about 17:20 without missing much).

Post by Steve Langasek
- abi-compliance-checker was used to analyze headers of 767 library
packages in the archive. Of those, 82 of 558 were successfully analyzed
and identified as library packages that would need patched for the
switch to 64-bit time_t to mark them as ABI-incompatible with the
previous version on armhf.
- an additional 209 packages could not be analyzed, because a-c-c failed
to compile the headers using its invocation of gcc. Assuming a roughly
equal distribution of ABI-changing vs non-ABI-changing libraries among
these that have failed to compile, we'd be looking at around 113 of 767
libraries that need changed.
- if we decide the level of effort to patch this many libraries is
worthwhile, to get an actual actionable list of libraries that need
patched for this transition would require getting the compile failures
down to zero. I did dig into the failures alphabetically from "a" to
"libe"; most of these failures were resolved by one of three methods.
- letting a-c-c know that this is a C library and to not try to compile
it with gcc's C++ frontend;
- excluding various headers that can't be included directly, either
because they are internal headers that are included via other headers
in the package, or because they have dependencies on other headers not
in the archive and can't be compiled (and therefore can't affect the
library ABI);
- pulling in additional undeclared package dependencies.
Even when applying these fixes, I still had 8 library packages whose
headers I wasn't able to get to compile. So there's still quite a bit
of work to do here.
- On this preliminary pass, I only included library packages that shipped
both headers and .so's, as this supposedly gives the best quality
results from a-c-c. To get a full list of ABI-breaking changes,
however, we need to include packages that ship only headers and not
.so's, to catch those libraries for which the .so is in a different
binary package (boost), and to catch plugin systems for executables
(apache).
While I'm currently working on this using Ubuntu, the results are largely
applicable to Debian, and the script itself certainly is. Note however that
there are a couple of bugs in abi-compliance-checker that I've patched in
Ubuntu, bug #951076 and bug #1030540, which you would want to have fixed
before trying to run this analysis in Debian. Cc:ing the a-c-c maintainer
for awareness of this.

I tried just running the script on debian and it tried 121 libraries
before dying, and produced compatibility results for 71. So yes it
needs some work to get useful numbers out.

You just ran this on Ubuntu main, so there will be quite a lot more
libraries in debian, I presume?

Post by Steve Langasek
Does doing an ABI transition of ~113 libraries seem tractable to folks?

It certainly provides evidence for the idea that this is not
completely intractable, which I think many people (including me)
worried was the case initially.

Post by Steve Langasek
If
there's interest in moving forward on this, I'd say the next step should be
to take it to debian-devel for broader discussion/signoff. I would also
move my scripts into a git repository that folks can contribute to -- adding
the necessary overrides for a-c-c for each library, so we can get an
authoritative list of ABI breakages, is very parallelizable.

Yes I think we should proceed with this analysis on debian to get a
better handle on just how many libraries we are looking at.

I would also still like suggestions/test cases for $stuff that we
thing will/might break _other_ than library ABI transitions, which we
at least know how to handle in general. So far there have not been
many things suggested, my list of tests I can run to check has zero
entries. That may be good sign (there just aren't that many), or it
might be a bad sign (no-one knows/is checking).

I have created a releasegoal page to discuss this transition in
general, and collect relevant info:
https://wiki.debian.org/ReleaseGoals/64bit-time

I think it would be useful do a bit more work on the ABI checking and
data-collecting (e.g. raspian input) before going to debian-devel, but
not wait more than another week or two, because people are already
switching, possibly without fully understanding the implications:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1026204


Wookey

[Arnd Bergmann]

Post by Wookey

Post by Steve Langasek
Does doing an ABI transition of ~113 libraries seem tractable to folks?

It certainly provides evidence for the idea that this is not
completely intractable, which I think many people (including me)
worried was the case initially.

When I did a similar analysis a few years ago using just pattern-matching
on header files, the result was that more than half the total packages
in Debian depended on at least one other package that needed an ABI
transition, which in my mind made it unrealistic.

It's possible that there were a lot of false-positive results in my
analysis then, but I would still suggest looking at what portion of
the archive depends on the libraries that you identified this time.

Arnd

[Diederik de Haas]

Post by Arnd Bergmann

Post by Wookey

Post by Steve Langasek
Does doing an ABI transition of ~113 libraries seem tractable to folks?

It certainly provides evidence for the idea that this is not
completely intractable, which I think many people (including me)
worried was the case initially.

When I did a similar analysis a few years ago using just pattern-matching
on header files, the result was that more than half the total packages
in Debian depended on at least one other package that needed an ABI
transition, which in my mind made it unrealistic.

If you do it in the early stage of Trixie's dev cycle, would it still be
unrealistic? It may be a bumpy ride and take a while, but I don't see an
immediate issue with that. Sid may finally become Unstable again ;-P

But *when* you do it, is quite relevant. If you/we are only a few months away
from the Trixie Freeze, then it's probably not a good idea.
But if we're 1-1.5 years before that, there's plenty of time to fix things.

Or is that too simplistic on my part?

[Wookey]

Post by Diederik de Haas

Post by Arnd Bergmann

Post by Wookey

Post by Steve Langasek
Does doing an ABI transition of ~113 libraries seem tractable to folks?

It certainly provides evidence for the idea that this is not
completely intractable, which I think many people (including me)
worried was the case initially.

When I did a similar analysis a few years ago using just pattern-matching
on header files, the result was that more than half the total packages
in Debian depended on at least one other package that needed an ABI
transition, which in my mind made it unrealistic.

If you do it in the early stage of Trixie's dev cycle, would it still be
unrealistic? It may be a bumpy ride and take a while, but I don't see an
immediate issue with that. Sid may finally become Unstable again ;-P
But *when* you do it, is quite relevant. If you/we are only a few months away
from the Trixie Freeze, then it's probably not a good idea.
But if we're 1-1.5 years before that, there's plenty of time to fix things.
Or is that too simplistic on my part?

No. I think that's reasonable.

One thing we do have to worry about right now is that anything which
has enabled LFS anytime in the last 15 years that rebuilds against a
current gnulib will automatically get 64-bit time_t unless they
explicitly turn it off. We may well have packages in the archive that
have had this done to them without noticing, and clearly we don't want
to do this in bookworm.

I'm not quite sure what the best way to check for this is. Is there a
place one can grep all buildlogs? AIUI gnulib is statically included
in the build a-la autoconf so it probably only happens when a new
upstream is pulled in with updated gnulib. BICBW.

gnulib doesn't seem to do releases (last tag 9 years ago 'v0.1') so I'm not quite sure when this changed.
Ah July 2021: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=history;f=m4/year2038.m4;h=2e4427e6fac10550c99748abebf31b61e6afda2b;hb=b9bf95fd0a6ab666b484b1e224321664f051f7fa

So the module has existed since 2017, but only defaults on since mid
2021. I'm not sure what one should be looking for in build logs to set
off the alarms.

Wookey

[Paul Wise]

Is there a place one can grep all buildlogs?

Debian members can access them on the buildd.debian.org server for eg:

/srv/buildd.debian.org/db/0/0ad/0.0.11-1/i386_1347540026_log.bz2

Please keep an eye on system load while searching them though.

If you only need build-dep versions, probably a better option is
the archive of buildinfo files and the database for searching it.

https://buildinfos.debian.net/ftp-master.debian.org/
https://salsa.debian.org/bremner/builtin-pho/

[Wookey]

Post by Wookey
Yes I think we should proceed with this analysis on debian to get a
better handle on just how many libraries we are looking at.

OK. We have over 10,000 *-dev package in debian so this took a while,
but I've now finished an initial pass.
The results are:
Total -dev packages: 10323
Time_t changes ABI: 329
LFS changes ABI: 58 0.6%
ABI unchanged: 1840 5.6%
Compilation failed: 1637
No headers in pkg: 3852
No library in pkg: 5086
total analysed: 10249
(note that the no-headers and no-libs categories are not exclusive
- the other classifications are)

Not quite sure where we lost 74 packages down the back of the sofa but at 0.7% it's noise.

So we have about 5000 packages which can basically be ignored for this purpose:
golang* (1943)
librust* (1955) - source-only, no dynamic linking, no .so's in any package
and libghc* (1065) - changes ABI at drop of hat (every new version) anyway.

Of the remaining 5360, 5237 have .so files to check. Of those 329
change ABI and another 58 are not built with LFS currently
so also change ABI due to that. That's 387 (7%).

34% are fine. And an annoyingly large 1637 (30%) did not build under
Abi-compliance-checker to determine one way or the other. Assuming 7%
of those are a problem that's another 114 packages.

I've yet to determine how many of the 'no-libs/no-headers' packages
actually expose an ABI we need to worry about. There will be some
more, from stuff like boost and Qt.

So the scale of the transition task in Debian is quite a lot bigger than in
Ubuntu: Probably 400-500 packages.

I've linked my lists on the wiki page.
https://wiki.debian.org/ReleaseGoals/64bit-time

I believe Steve L (who has done most of the heavy lifting on the
script) is running a check too and will have some results soon.

Wookey

[Steve Langasek]

Sorry for the long radio silence on this; I had identified some issues with
my prior report and wanted to rerun it with some fixes, and that analysis
took rather longer than expected (mainly due to infrastructure slowness).

Refreshed output for Ubuntu is here:

https://people.canonical.com/~vorlon/armhf-time_t/

So comparing with Wookey's results, I see:

Total -dev packages tested: 4603
Time_t changes ABI: 349
LFS changes ABI: 71
ABI unchanged: 2085
Compilation failed: 2098

So certainly this appears to be the same order of magnitude as Wookey's
separate analysis.

The other question is what impact this has on the archive as a whole, with
triggering of rebuild transitions. Looking at the
reverse-build-dependencies of those 349 packages gives:

$ while read pkg; do grep-dctrl -n -sPackage -w -FBuild-Depends $pkg /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lunar_*Sources ; done < abi-breaks | sort -u | wc -l
3626
$

NB this is a bit of an undercount as a result of packages that ship headers
but aren't the thing being directly build-depended on. A more accurate
count is probably to look at binary packages, from the same source as the
-dev package, that the -dev package depends on and look at
reverse-dependencies of those. But for a first pass, here's where we are.

There are also roughly 2100 packages for which compilation failed. To know
which packages actually have ABI changes and require transitions, we would
need to work through this list and get the count down to 0.

But as a first-order approximation, this looks doable to me as a transition
we could manage.

What do other folks think? Are you on board with proposing this to
debian-devel? Anyone want to help add the necessary quirks to drive down
the list of compilation failures?

Post by Wookey

Post by Wookey
Yes I think we should proceed with this analysis on debian to get a
better handle on just how many libraries we are looking at.

OK. We have over 10,000 *-dev package in debian so this took a while,
but I've now finished an initial pass.
Total -dev packages: 10323
Time_t changes ABI: 329
LFS changes ABI: 58 0.6%
ABI unchanged: 1840 5.6%
Compilation failed: 1637
No headers in pkg: 3852
No library in pkg: 5086
total analysed: 10249
(note that the no-headers and no-libs categories are not exclusive
- the other classifications are)
Not quite sure where we lost 74 packages down the back of the sofa but at 0.7% it's noise.
golang* (1943)
librust* (1955) - source-only, no dynamic linking, no .so's in any package
and libghc* (1065) - changes ABI at drop of hat (every new version) anyway.
Of the remaining 5360, 5237 have .so files to check. Of those 329
change ABI and another 58 are not built with LFS currently
so also change ABI due to that. That's 387 (7%).
34% are fine. And an annoyingly large 1637 (30%) did not build under
Abi-compliance-checker to determine one way or the other. Assuming 7%
of those are a problem that's another 114 packages.
I've yet to determine how many of the 'no-libs/no-headers' packages
actually expose an ABI we need to worry about. There will be some
more, from stuff like boost and Qt.
So the scale of the transition task in Debian is quite a lot bigger than in
Ubuntu: Probably 400-500 packages.
I've linked my lists on the wiki page.
https://wiki.debian.org/ReleaseGoals/64bit-time
I believe Steve L (who has done most of the heavy lifting on the
script) is running a check too and will have some results soon.
Wookey
--
Principal hats: Debian, Wookware, ARM
http://wookware.org/

[Steve Langasek]

Post by Steve Langasek
Sorry for the long radio silence on this; I had identified some issues with
my prior report and wanted to rerun it with some fixes, and that analysis
took rather longer than expected (mainly due to infrastructure slowness).
https://people.canonical.com/~vorlon/armhf-time_t/
Total -dev packages tested: 4603
Time_t changes ABI: 349
LFS changes ABI: 71
ABI unchanged: 2085
Compilation failed: 2098
So certainly this appears to be the same order of magnitude as Wookey's
separate analysis.

A little more digging here into the failures; I've prioritized them by
number of reverse-build-depends:

while read devpkg; do
count=$(grep-dctrl -w -FBuild-Depends $devpkg -n -sPackage \
/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lunar_*Sources \
| sort -u | wc -l)
echo "$count $devpkg"; done < failed-compilation \
| sort -n -r > sorted-revdep-count

One nice finding is that 807 of the libraries that have failed analysis have
no reverse-build-dependencies at all, so can be more or less ignored for the
transition (we should still follow through on the analysis and rename
packages, but this can happen any time during the trixie release cycle).

One *unfortunate* finding, having fixed up the compilation of the top 5
failures, is that each of libssl-dev, libgtk-3-dev, and libgtk2.0-dev breaks
ABI with 64-bit time_t.

Post by Steve Langasek
The other question is what impact this has on the archive as a whole, with
triggering of rebuild transitions. Looking at the
$ while read pkg; do grep-dctrl -n -sPackage -w -FBuild-Depends $pkg /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lunar_*Sources ; done < abi-breaks | sort -u | wc -l
3626
$

But on the bright side again, this only increases the number of revdeps that
are part of the transition from 3626 to 4004, due to overlap with other
libraries.

From my perspective, this is entirely achievable and we should move forward
with discussing this more broadly on debian-devel.

Note that while armhf is the priority wrt Ubuntu, this would also address
2038 compatibility for armel and mipsel as Debian release archs. (I do not
think we should change i386 to 64-bit time_t as the primary value of this
arch is for binary compatibility with legacy applications that can't be
recompiled.)

Post by Steve Langasek
NB this is a bit of an undercount as a result of packages that ship headers
but aren't the thing being directly build-depended on. A more accurate
count is probably to look at binary packages, from the same source as the
-dev package, that the -dev package depends on and look at
reverse-dependencies of those. But for a first pass, here's where we are.
There are also roughly 2100 packages for which compilation failed. To know
which packages actually have ABI changes and require transitions, we would
need to work through this list and get the count down to 0.
But as a first-order approximation, this looks doable to me as a transition
we could manage.
What do other folks think? Are you on board with proposing this to
debian-devel? Anyone want to help add the necessary quirks to drive down
the list of compilation failures?

Post by Wookey

Post by Wookey
Yes I think we should proceed with this analysis on debian to get a
better handle on just how many libraries we are looking at.

OK. We have over 10,000 *-dev package in debian so this took a while,
but I've now finished an initial pass.
Total -dev packages: 10323
Time_t changes ABI: 329
LFS changes ABI: 58 0.6%
ABI unchanged: 1840 5.6%
Compilation failed: 1637
No headers in pkg: 3852
No library in pkg: 5086
total analysed: 10249
(note that the no-headers and no-libs categories are not exclusive
- the other classifications are)
Not quite sure where we lost 74 packages down the back of the sofa but at 0.7% it's noise.
golang* (1943)
librust* (1955) - source-only, no dynamic linking, no .so's in any package
and libghc* (1065) - changes ABI at drop of hat (every new version) anyway.
Of the remaining 5360, 5237 have .so files to check. Of those 329
change ABI and another 58 are not built with LFS currently
so also change ABI due to that. That's 387 (7%).
34% are fine. And an annoyingly large 1637 (30%) did not build under
Abi-compliance-checker to determine one way or the other. Assuming 7%
of those are a problem that's another 114 packages.
I've yet to determine how many of the 'no-libs/no-headers' packages
actually expose an ABI we need to worry about. There will be some
more, from stuff like boost and Qt.
So the scale of the transition task in Debian is quite a lot bigger than in
Ubuntu: Probably 400-500 packages.
I've linked my lists on the wiki page.
https://wiki.debian.org/ReleaseGoals/64bit-time
I believe Steve L (who has done most of the heavy lifting on the
script) is running a check too and will have some results soon.
Wookey
--
Principal hats: Debian, Wookware, ARM
http://wookware.org/

[Steve Langasek]

Post by Wookey

Post by Steve Langasek
This is good. One thing that I think has been missing from the discussion
about armhf rebootstrap is the fact that we do have experience in Debian
doing cross-cutting ABI transitions without having to change an
the libc5->libc6 transition ('g' suffix - which still remains today in
libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long
double' transition ('ldbl' suffix, and an example of doing this for an ABI
transition that didn't affect all architectures).

I feel like I should already know this, but after a bit of thought,
https://wiki.debian.org/Teams/ReleaseTeam/Transitions I do not
understand how the suffixes are used in these transitions.
That wiki page does not appear to mention adding suffixes to package
names. It appears to only document the process where the version is
just bumped, everything involved is rebuilt in experimental and then
re-uploaded to unstable once we are ready to make the change.
I presume that the suffix is used for transitions involving a large
number of packages, in order to keep both the old-ABI and new-ABI
version of the packages around in parallel (and conflicting with each
other?) for a while which is necessary for larger transitions because
doing it all in one go would take too long, or get entangled with
uploads from the unware that might cause breakage, or something?

When we have an ABI change in the distro that affects shared libraries, but
is not the result of sourceful changes to the upstream source, we don't want
to change the soname of the library; that's upstream's domain.

But we do need at the packaging level to distinguish between two packages of
the library that provide different ABIs, so that packages depending on one
ABI don't incorrectly get the package with the other ABI, resulting in
broken runtime behavior.

When we transition, the un-annotated package will remain present in older
releases, but will be entirely superseded in unstable by the package
providing the new ABI.

BTW, I'll suggest "t64" as the library name suffix here.

Post by Wookey
Just rebuilding and not renaming sounds like a much smoother process,
because as soon as we rename then there is a load of renaming in
reverse dependencies too, and there is presumably a set of rules about
when conflicts are added and other details that need to be got right.

No, that breaks partial upgrades and makes a right mess of unstable *and* of
upgrades between stable releases. "Just rebuilding and not renaming" is the
"rebootstrap" approach which I am hoping to avoid.

[Aurelien Jarno]

Hi,

Post by Helmut Grohne
Attempt #2: reproducible
My other favourite QA tool is reproducible builds. So how about building
and comparing binary packages?
Unfortunately, build flags participate in build ids. That directly
implies a high risk of false positives and/or manual work.
Without further ado, here we go.
* base-files
* db-defaults
* coreutils

coreutils 9.1 enables y2038 support by default if available, so this is
actually expected.

Regards
Aurelien