Status of dpkg-shlibdeps tracking ARM object linkage ABI mismatches

[Guillem Jover]

Hi Steve!

I was recently working on the Dpkg::Shlibs::Objdump module code
related to ELF and ABI tracking, and when seeing the ARM handling
missing there, recalled the issues we saw some time ago with ARM
when I tried to make that tracking more strict, which had to be
reverted due to issues with objects in the wild. For context that
was #853793.

I was wondering whether you (or know if someone else) had worked on
the ARM port side of things to clean up those issues, from toolchain to
specific objects in packages? I'm not in a hurry to add that code back
so do not feel any pressure from this, I'm mostly wondering where we are
at with this, and whether there is something I can improve from dpkg side
in that regard, but if not that's also fine, and then I'd probably try
to update the status somewhere (code comment or wiki or something).

(I think at least the issue with wine should be solved now with commit
https://salsa.debian.org/wine-team/wine/-/commit/51f48d3e6c04cef760610d14ba5f368e7f2baf7a)

Thanks,
Guillem

[Steve McIntyre]

Hey Guillem!

Post by Guillem Jover
Hi Steve!
I was recently working on the Dpkg::Shlibs::Objdump module code
related to ELF and ABI tracking, and when seeing the ARM handling
missing there, recalled the issues we saw some time ago with ARM
when I tried to make that tracking more strict, which had to be
reverted due to issues with objects in the wild. For context that
was #853793.

Oh $deity, that's going back a while... :-)

Post by Guillem Jover
I was wondering whether you (or know if someone else) had worked on
the ARM port side of things to clean up those issues, from toolchain to
specific objects in packages? I'm not in a hurry to add that code back
so do not feel any pressure from this, I'm mostly wondering where we are
at with this, and whether there is something I can improve from dpkg side
in that regard, but if not that's also fine, and then I'd probably try
to update the status somewhere (code comment or wiki or something).

I've moved on from being employed by Arm 3 years ago, and I've got
plenty of other things to keep me busy now. If we're still seeing
issues in packages today, then maybe we might find some help from
Wookey or Emmanuel (who should both be reading this list!).

Post by Guillem Jover
(I think at least the issue with wine should be solved now with commit
https://salsa.debian.org/wine-team/wine/-/commit/51f48d3e6c04cef760610d14ba5f368e7f2baf7a)

Nod.

[Wookey]

Post by Steve McIntyre
If we're still seeing
issues in packages today, then maybe we might find some help from
Wookey or Emmanuel (who should both be reading this list!).

I am, and have noticed this issue and put it on our list of things to look
at. I was going to wait until I had something useful to say before
replying :-)

Wookey

[Guillem Jover]

Hi!

Post by Wookey

Post by Steve McIntyre
If we're still seeing
issues in packages today, then maybe we might find some help from
Wookey or Emmanuel (who should both be reading this list!).

I am, and have noticed this issue and put it on our list of things to look
at. I was going to wait until I had something useful to say before
replying :-)

Ah, great! Thanks both for the update. Then I think I'll just add a
comment on the code for now with a ref to that old revert, and it can
be refreshed or acted on once there are news on this front, so that I
do not forget about this.

Thanks,
Guillem

[Emanuele Rocca]

Hi Guillem,

Post by Guillem Jover
I was recently working on the Dpkg::Shlibs::Objdump module code
related to ELF and ABI tracking, and when seeing the ARM handling
missing there, recalled the issues we saw some time ago with ARM
when I tried to make that tracking more strict, which had to be
reverted due to issues with objects in the wild. For context that
was #853793.

My understanding is that the problematic cases are ELF files with:

(1) EABIv4 instead of EABIv5
(2) armhf files with the ARM_SOFT_FLOAT flag set
(3) armel files with the ARM_HARD_FLOAT flag set

Is that correct? Anything else?

Thanks!
Emanuele

[Guillem Jover]

Hi!

Post by Emanuele Rocca

Post by Guillem Jover
I was recently working on the Dpkg::Shlibs::Objdump module code
related to ELF and ABI tracking, and when seeing the ARM handling
missing there, recalled the issues we saw some time ago with ARM
when I tried to make that tracking more strict, which had to be
reverted due to issues with objects in the wild. For context that
was #853793.

(1) EABIv4 instead of EABIv5
(2) armhf files with the ARM_SOFT_FLOAT flag set
(3) armel files with the ARM_HARD_FLOAT flag set
Is that correct? Anything else?

AFAIR there was also the case of objects being annotated with
Tag_ABI_VFP_args but not with either of the ABI hard or soft float
flags. And rechecking the commit message, it seems there were also
objects with both ABI float flags set at the same time.

I'm not sure whether you plan on analyzing all ELF objects in the
Debian archive for the Arm architectures, but if so, that would be
rather helpful. But thanks in any case for looking into this!

Thanks,
Guillem

[Emanuele Rocca]

Hey,

Post by Guillem Jover
AFAIR there was also the case of objects being annotated with
Tag_ABI_VFP_args but not with either of the ABI hard or soft float
flags. And rechecking the commit message, it seems there were also
objects with both ABI float flags set at the same time.
I'm not sure whether you plan on analyzing all ELF objects in the
Debian archive for the Arm architectures, but if so, that would be
rather helpful. But thanks in any case for looking into this!

I did a first pass, and it seems that the only objects with Version4 are
generated by tcc.

Flags according to readelf -h on armel:
0x4000200, Version4 EABI, <unknown>

And on armhf:
0x4000400, Version4 EABI, <unknown>

For reference, the files are:

/usr/lib/arm-linux-gnueabi{,hf}/tcc/bcheck.o
/usr/lib/arm-linux-gnueabi{,hf}/tcc/bt-exe.o
/usr/lib/arm-linux-gnueabi{,hf}/tcc/bt-log.o

Those are probably not particularly interesting for dpkg-shlibdeps, but
I've filed #1038162 nonetheless.

You mentioned in #853793 paris-traceroute also targeting Version4, but
that package is now gone.

I'll look at the soft/hard float ones next.

Thanks,
ema

[Emanuele Rocca]

Post by Emanuele Rocca
I'll look at the soft/hard float ones next.

Two findings.

(1) I couldn't find any armel object with the hard-float flag set.

(2) There are a few armhf packages shipping files with the soft-float flag.

All of them, with the exception of the u-boot packages, are written in
Pascal. It seems that fpc just emits the wrong flags. As an example,
here is the readelf output for the armhf version of cqrlog. Note that
Tag_ABI_VFP_args is not set either.

$ readelf -A /usr/bin/cqrlog
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "7-A"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_FP_arch: VFPv3-D16
Tag_ABI_align_needed: 8-byte

$ readelf -h /usr/bin/cqrlog
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: ARM
Version: 0x1
Entry point address: 0x28620
Start of program headers: 52 (bytes into file)
Start of section headers: 15683592 (bytes into file)
Flags: 0x5000200, Version5 EABI, soft-float ABI
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 8
Size of section headers: 40 (bytes)
Number of section headers: 26
Section header string table index: 25

Full list of armhf packages shipping ELF objects with the soft-float
flag set:

astap
astap-cli
c-evo-dh-gtk2
c-evo-dh-stdai
cqrlog
doublecmd-gtk
doublecmd-plugins
doublecmd-qt
el-ixir
fp-compiler-3.2.2
fp-ide-3.2.2
fp-units-castle-game-engine
fp-units-rtl-3.2.2
fp-utils-3.2.2
gearhead
gearhead-sdl
goverlay
lazpaint-gtk2
lazpaint-qt5
mricron
nbc
pasdoc
tomboy-ng
u-boot-rockchip
u-boot-rpi
u-boot-sunxi
u-boot-tegra
view3dscene
winff-gtk2
winff-qt

[Vagrant Cascadian]

Post by Emanuele Rocca
(1) I couldn't find any armel object with the hard-float flag set.
(2) There are a few armhf packages shipping files with the soft-float flag.
All of them, with the exception of the u-boot packages, are written in
Pascal. It seems that fpc just emits the wrong flags. As an example,
here is the readelf output for the armhf version of cqrlog. Note that
Tag_ABI_VFP_args is not set either.

...

Post by Emanuele Rocca
Full list of armhf packages shipping ELF objects with the soft-float

...

Post by Emanuele Rocca
u-boot-rockchip
u-boot-rpi
u-boot-sunxi
u-boot-tegra

For u-boot, the boot process for some boards may include components that
are running baremetal and are not strictly armhf-capable, so that is not
hugely surprising...

Maybe there are some performance gains to be hard, but as long as u-boot
can load a kernel and successfully boot into userspace on these
platforms, I would not be too worried about weather hard-float is
enabled for these targets.

live well,
vagrant

[Emanuele Rocca]

Hi,

Post by Guillem Jover
AFAIR there was also the case of objects being annotated with
Tag_ABI_VFP_args but not with either of the ABI hard or soft float
flags.

Indeed, there are 1 armel and 91 armhf binary packages shipping ELF
files without float flags (hard or soft) but with TAG_ABI_VFP_ARGS.

Examples:

gcc-arm-none-eabi_12.2.rel1-1_armel.deb ./usr/lib/gcc/arm-none-eabi/12.2.1/arm/v5te/hard/crtbegin.o 0x5000000: Version5 EABI TAG_ABI_VFP_ARGS

clisp-module-fastcgi_2.49.20210628.gitde01f0f-3_armhf.deb ./usr/lib/clisp-2.49.93+/fastcgi/fastcgi.o 0x5000000: Version5 EABI TAG_ABI_VFP_ARGS

What do we want to do about those? I can take care of filing bug reports
asking to add the missing flags if we agree it's a good idea.

Post by Guillem Jover
And rechecking the commit message, it seems there were also
objects with both ABI float flags set at the same time.

I couldn't find any such case in my analysis, do you perhaps remember
which objects were affected?

Thanks,
Emanuele

[Guillem Jover]

Hi!

Post by Emanuele Rocca
I did a first pass, and it seems that the only objects with Version4 are
generated by tcc.
0x4000200, Version4 EABI, <unknown>
0x4000400, Version4 EABI, <unknown>
/usr/lib/arm-linux-gnueabi{,hf}/tcc/bcheck.o
/usr/lib/arm-linux-gnueabi{,hf}/tcc/bt-exe.o
/usr/lib/arm-linux-gnueabi{,hf}/tcc/bt-log.o
Those are probably not particularly interesting for dpkg-shlibdeps, but
I've filed #1038162 nonetheless.

Thanks, and right I guess as long as no package builds using tcc and
we do not end up with those in the archive, then that seems fine. But
in any case it would indeed be nice to get fixed.

Post by Emanuele Rocca
You mentioned in #853793 paris-traceroute also targeting Version4, but
that package is now gone.

Ah, good then, I guess. :)

Post by Emanuele Rocca

Post by Emanuele Rocca
I'll look at the soft/hard float ones next.

Two findings.
(1) I couldn't find any armel object with the hard-float flag set.

Great!

Post by Emanuele Rocca
(2) There are a few armhf packages shipping files with the soft-float flag.
All of them, with the exception of the u-boot packages,

I agree with Vagrant, that this does not seem problematic, and I guess
strictly speaking boot loaders would instead be packaged as part of
some kernel-less architecture, but because we are not there now (or
ever), then this seems fine.

Post by Emanuele Rocca
are written in
Pascal. It seems that fpc just emits the wrong flags. As an example,
here is the readelf output for the armhf version of cqrlog. Note that
Tag_ABI_VFP_args is not set either.

Oh, hmm, could you file a report for this (perhaps preferably
upstream)? (Or perhaps you did already? :)

Post by Emanuele Rocca
Full list of armhf packages shipping ELF objects with the soft-float
astap
astap-cli
c-evo-dh-gtk2
c-evo-dh-stdai
cqrlog
doublecmd-gtk
doublecmd-plugins
doublecmd-qt
el-ixir
fp-compiler-3.2.2
fp-ide-3.2.2
fp-units-castle-game-engine
fp-units-rtl-3.2.2
fp-utils-3.2.2
gearhead
gearhead-sdl
goverlay
lazpaint-gtk2
lazpaint-qt5
mricron
nbc
pasdoc
tomboy-ng

[…]

Post by Emanuele Rocca
view3dscene
winff-gtk2
winff-qt

Hmm I guess these are going to be problematic for dpkg-shlibdeps when
trying to analyze these objects against the shared libraries they link
against.

Post by Emanuele Rocca

Post by Emanuele Rocca
AFAIR there was also the case of objects being annotated with
Tag_ABI_VFP_args but not with either of the ABI hard or soft float
flags.

Indeed, there are 1 armel and 91 armhf binary packages shipping ELF
files without float flags (hard or soft) but with TAG_ABI_VFP_ARGS.
gcc-arm-none-eabi_12.2.rel1-1_armel.deb ./usr/lib/gcc/arm-none-eabi/12.2.1/arm/v5te/hard/crtbegin.o 0x5000000: Version5 EABI TAG_ABI_VFP_ARGS

Given that this looks like a cross toolchain, it might need to be
exempted, as these might end up shipping objects for the target arch
which will not match the package arch.

Post by Emanuele Rocca
clisp-module-fastcgi_2.49.20210628.gitde01f0f-3_armhf.deb ./usr/lib/clisp-2.49.93+/fastcgi/fastcgi.o 0x5000000: Version5 EABI TAG_ABI_VFP_ARGS

This seems like something to fix.

Post by Emanuele Rocca
What do we want to do about those? I can take care of filing bug reports
asking to add the missing flags if we agree it's a good idea.

I guess it would be interesting to know whether this is due to these
having been built with old toolchains (and never been rebuilt since),
or there's some toolchain or build machinery issue at play or similar.
Otherwise it would be nice to get consistency here, and I'm in
particular looking forward to being able to re-enable the ABI checker
in dpkg-shlibdeps. :)

Post by Emanuele Rocca

Post by Emanuele Rocca
And rechecking the commit message, it seems there were also
objects with both ABI float flags set at the same time.

I couldn't find any such case in my analysis, do you perhaps remember
which objects were affected?

Rechecking the bug report, it looks like this was the part with
sonames2elf from wine, which is supposedly fixed now. But it seems
there was a potential problem with gcc mishandling builds that did not
include libc, see this message which describes this a bit more:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=853793#45

where I don't know whether that part got reported or is fixed in gcc.

And thanks for doing this analysis! Much appreciated.

Regards,
Guillem

[Emanuele Rocca]

Hi,

Post by Guillem Jover

Post by Emanuele Rocca
are written in
Pascal. It seems that fpc just emits the wrong flags. As an example,
here is the readelf output for the armhf version of cqrlog. Note that
Tag_ABI_VFP_args is not set either.

Oh, hmm, could you file a report for this (perhaps preferably
upstream)? (Or perhaps you did already? :)

Done, see https://gitlab.com/freepascal.org/fpc/source/-/issues/40374
and https://bugs.debian.org/1038868.

Post by Guillem Jover

Post by Emanuele Rocca
astap
astap-cli
c-evo-dh-gtk2
c-evo-dh-stdai
cqrlog
doublecmd-gtk
doublecmd-plugins
doublecmd-qt
el-ixir
fp-compiler-3.2.2
fp-ide-3.2.2
fp-units-castle-game-engine
fp-units-rtl-3.2.2
fp-utils-3.2.2
gearhead
gearhead-sdl
goverlay
lazpaint-gtk2
lazpaint-qt5
mricron
nbc
pasdoc
tomboy-ng

[…]

Post by Emanuele Rocca
view3dscene
winff-gtk2
winff-qt

Hmm I guess these are going to be problematic for dpkg-shlibdeps when
trying to analyze these objects against the shared libraries they link
against.

Once fpc is fixed, we could maybe rebuild all of the above?

Post by Guillem Jover

Post by Emanuele Rocca
clisp-module-fastcgi_2.49.20210628.gitde01f0f-3_armhf.deb ./usr/lib/clisp-2.49.93+/fastcgi/fastcgi.o 0x5000000: Version5 EABI TAG_ABI_VFP_ARGS

This seems like something to fix.

Post by Emanuele Rocca
What do we want to do about those? I can take care of filing bug reports
asking to add the missing flags if we agree it's a good idea.

I guess it would be interesting to know whether this is due to these
having been built with old toolchains (and never been rebuilt since),
or there's some toolchain or build machinery issue at play or similar.

Nope, I've rebuilt clisp on armhf and the flags are still wrong. I'll
look into this further.

Post by Guillem Jover
Otherwise it would be nice to get consistency here

We'll get there! :)

[Emanuele Rocca]

Hi,

Post by Emanuele Rocca

Post by Guillem Jover
AFAIR there was also the case of objects being annotated with
Tag_ABI_VFP_args but not with either of the ABI hard or soft float
flags.

Indeed, there are 1 armel and 91 armhf binary packages shipping ELF
files without float flags (hard or soft) but with TAG_ABI_VFP_ARGS.
gcc-arm-none-eabi_12.2.rel1-1_armel.deb ./usr/lib/gcc/arm-none-eabi/12.2.1/arm/v5te/hard/crtbegin.o 0x5000000: Version5 EABI TAG_ABI_VFP_ARGS
clisp-module-fastcgi_2.49.20210628.gitde01f0f-3_armhf.deb ./usr/lib/clisp-2.49.93+/fastcgi/fastcgi.o 0x5000000: Version5 EABI TAG_ABI_VFP_ARGS

It turns out that the above is correct. The hard/soft flag is set by the
linker only on executables or shared libraries, not on relocatable
objects (such as the ones above generated by gcc -c).

Different relocatable objects, when linked together, may have different
build attributes. They could even be conflicting (eg: hard-float with
soft-float) and lead to linker errors such as:

ld: error: b.o uses VFP register arguments, a.out does not

Bottom-line, I should exclude relocatable objects from my analysis and
only consider final, linked images.